Adam Smith, the Monumental Failure of Cybersecurity, and ‘Rogue Agents’. What Next?

A software maker takes the revenue from its products while its customers pay to secure them, often uncovering yet more vulnerabilities in the process. AI is now making the consequences of this bizarre arrangement urgent as weaknesses are exposed at unprecedented scale. At the same time, cyber-attacks and defences are turning into a machine-speed arms race.

A new lexicon has emerged around AI agents, autonomy and unexpected behaviour that risks obscuring the escalating threat we all face. Software is escaping controls, exploiting vulnerabilities and gaining unauthorised access to systems. The term often given to this software is ‘rogue agent’. This is deeply misleading. We are dealing with malware, plain and simple.

Adam Smith would have identified this as an extremely imperfect market. In this article, I turn to Smith to build a central argument about the cybersecurity market and the incentives that distort it.

In 1776, Adam Smith wrote in ‘The Wealth of Nations’ that ‘it is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest.’ People act on the incentives in front of them. A software maker that can transfer much of the cost of insecurity and safety to its customers has weaker incentives to eliminate it at source, and a security industry paid to manage insecurity grows as insecurity grows. AI is now accelerating that vicious cycle as a new generation of autonomous software creates new risks and another generation of security products emerges to contain them.

The result is familiar to every board. Organisations with large security budgets, talented teams and sophisticated tools are still breached, leaving CISOs and their teams playing whack-a-mole as they find the flaw, patch it, add another control and buy another product before the next crisis.

Security teams stop attacks every day and individual controls often work extremely well, but we have still ended up with a huge industry profiting from insecure technology while customers pick up the bill for its shortcomings.

This is not an argument that cybersecurity is pointless. I am arguing that the current model is increasingly unsustainable in the face of an AI and nomenclature onslaught. There is no single fix. Regulation, procurement, insurance, liability and better software engineering can all play a part. But changing the incentives is necessary. Software suppliers need to carry more responsibility for the security of the products they sell, with economic and regulatory consequences when they fail to do so. In a properly functioning market, customers would not have to buy layer after layer of security to compensate for weaknesses in technology they have already purchased. Building secure products should be more profitable than profiting from their insecurity.

As AI takes over more detection, triage and response work, the CISO role will collapse into risk, resilience and AI governance, with far less emphasis on running a large AI-enabled security operation.

The whack-a-mole bargain

An enterprise buys software and inherits much of the work of securing it. It configures and patches the product, manages identities, monitors activity, segments the network, buys endpoint protection and vulnerability scanning, hires specialists, commissions penetration tests and takes out cyber insurance. If the product is compromised, the enterprise pays for the investigation, answers to the regulator, faces its board and explains itself to its customers, before returning to the cybersecurity game of whack-a-mole and spending even more on security.

Few buyers would accept a car on the understanding that another industry would fit the brakes and airbags later, then spend years searching for and fixing safety problems the manufacturer had missed. Software is more complicated, will always contain flaws and is constantly being tested by people trying to break it, so secure-by-design will never mean eliminating every vulnerability. But complexity has also become an excuse for a weird arrangement in which the supplier builds the product and the customer pays much of the cost when its security fails.

AI makes this arrangement harder to defend. Software companies are releasing autonomous products that can escape controls and take unauthorised actions, bringing security and AI safety together. The term ‘rogue agent’ risks allowing software makers to abdicate yet more responsibility for the software they created and released.

Rogue agents are malware

AI has made the cybersecurity risk a much more critical issue. It is taking a sledgehammer to existing controls by exposing vulnerabilities at a scale and speed we have not seen before. Many of these weaknesses have been lurking in systems and networks for years but were expensive and time-consuming to find. AI is ruthlessly and relentlessly exposing the failure of a cybersecurity business model that allowed so many vulnerabilities to accumulate in the first place.

Security vendors are responding to AI-enabled attacks with AI-enabled defences that can find flaws, triage alerts and accelerate patching. Work that once required scarce skills and time can increasingly be carried out by machines at enormous scale. Attackers can find vulnerabilities faster, defenders can find and patch them faster, and attackers can move almost immediately to the next open goal. The cybersecurity arms race is moving to machine speed while the business model behind it remains largely unchanged.

Autonomous software makes the problem more serious. AI agents are escaping controls, exploiting vulnerabilities and gaining unauthorised access to systems. This is occurring with alarming frequency and appears to be turning into a new norm. OpenAI, Anthropic and other AI companies are struggling to demonstrate that their safeguards can keep pace with increasingly capable agents.

We also need to be much less precious about what to call these systems when they go rogue. Software that escapes its authorised environment, exploits vulnerabilities and gains unauthorised access to somebody else’s systems is malware, plain and simple. Human beings still control the levers that create and release this software. The rapidly expanding lexicon around autonomous AI risks making behaviour we already understand in cybersecurity sound like an entirely new category of problem.

This makes the cybersecurity crisis more acute. Customers already spend heavily protecting themselves against vulnerabilities in the technology they buy. Autonomous software is now becoming the security threat itself. Software companies take the risks while customers pay to contain the consequences, deepening the distortion that already exists in the cybersecurity market.

The cybersecurity industry is responding with another generation of products designed to discover agents, monitor their behaviour, control their access and stop unauthorised actions. Much of this technology will be necessary, but the economics are extraordinary. Software companies release increasingly powerful autonomous systems, those systems create new security risks, cybersecurity companies develop products to contain them and customers pay the bill. The market grows again.

Supplier flaws and customer mistakes

Customers create plenty of their own security problems such as misconfigured cloud storage, excessive privileges, weak passwords, unmanaged devices and critical patches left unapplied for months. These are problems only the customer can address, so responsibility for them will always remain with the organisation running the technology.

Meanwhile, suppliers are not taking enough responsibility for the security and safety of the technology they build and sell. Coding flaws, insecure defaults, vulnerable components, inadequate patching and short support periods result from decisions made by those suppliers. AI turbocharges this ongoing trend. Software is no longer a product containing vulnerabilities that somebody else might exploit. Autonomous software can itself escape controls, exploit vulnerabilities and gain unauthorised access to other systems. Software makers continue to build and release it. Using the term rogue agent does not make malware intrusion an act of God.

Customers cannot repair code they do not control or redesign an AI system they did not build. They can protect themselves against some of the consequences, but they end up paying heavily when a vulnerability is exploited or autonomous software behaves in ways its developer failed to contain.

Governments, regulators and cybersecurity agencies have been pushing secure-by-design and greater supplier responsibility for years, yet the economics have barely changed. AI safety now brings the same underlying problem into much sharper focus. Customers still pay to protect themselves from weaknesses in products they have already purchased and increasingly from the behaviour of autonomous software they did not create. Until suppliers have significantly more financial exposure to the insecurity and unsafe behaviour they introduce, there will be little economic incentive to eliminate it at source.

The platform that sells the protection

There is another peculiarity in today’s market. Some of the world’s largest technology companies are also among its largest cybersecurity companies. Microsoft is the clearest example, supplying operating systems, cloud infrastructure, identity, productivity software and AI while also selling security products to protect customers using those environments.

Customers can therefore pay a supplier for the platform and then pay the same supplier again for additional products to protect it. This creates a potential conflict of interest. The supplier benefits from making its technology more secure but can also earn additional revenue from selling the products needed to protect it.

The 2023 compromise of Microsoft Exchange Online showed the problem particularly well. The US State Department detected the intrusion using logging capabilities available through Microsoft’s premium G5 licence, while customers without the premium service lacked the same visibility. Following the incident and pressure from the US government, Microsoft made a wider range of cloud security logs available to customers at no additional cost. The incentives changed and so did the product.

Smith’s argument about self-interest applies directly. A properly functioning market would make secure technology more profitable than selling additional products to compensate for its weaknesses.

Changing the incentives

Governments have been urging suppliers to build in security for years. The United States developed secure software standards, introduced supplier attestations and launched CISA’s voluntary Secure by Design Pledge. But regulatory pressure on software suppliers in the US has weakened. The June 2025 executive order retained work on secure software development while removing provisions intended to use federal procurement and supplier attestations to exert greater pressure on software vendors. Guidance, standards and voluntary commitments remain, but they do not fundamentally change the economics. Suppliers still make the product while customers carry much of the cost when its security fails.

Europe is moving in the opposite direction. The revised Product Liability Directive explicitly covers software and AI systems, while the Cyber Resilience Act places cybersecurity obligations on manufacturers throughout the life of digital products. Both increase the potential legal and financial consequences for suppliers when their products cause harm.

This becomes much more significant as autonomous software spreads. If an AI agent escapes its controls, exploits a vulnerability and causes damage, the company that built and released the software should have greater financial exposure to the consequences.

Enterprise buyers can exert pressure too. Contracts can require important security capabilities to be included in the product, set expectations for fixing critical vulnerabilities and allocate more of the cost of supplier-controlled failures back to the supplier. Insurers can reinforce those incentives by taking greater account of suppliers’ security records and the behaviour of their AI systems.

Procurement also needs to give the companies supplying critical technology a harder time. Buyers need to ask how much of their cybersecurity spending protects them from their own mistakes and how much compensates for weaknesses in the technology they buy. Increasingly, they also need to ask how much they are spending to protect themselves from autonomous software somebody else built and released.

A larger and increasingly distorted market

For decades, cybersecurity has focused on defending organisations against ever more capable attackers, and a vast industry has grown around that function. AI is about to make that industry much bigger. Attack and defence are moving to machine speed, vulnerabilities can be found at unprecedented scale and autonomous software creates another category of risk that organisations will have to secure.

Cybersecurity vendors will respond because their customers need them to. Enterprises will spend more on protecting AI systems, controlling agents, managing machine identities, monitoring autonomous behaviour and defending themselves against AI-enabled attacks. Much of that spending will be necessary and many of those products will work.

But the growth of the cybersecurity industry should not be confused with the successful functioning of the market. If insecure technology creates demand for security products, and autonomous software creates another wave of products needed to contain it, the market can grow rapidly while the underlying problem gets worse. All of the participants are responding rationally to market incentives, even as the market itself becomes increasingly distorted.

For decades we have asked customers to get better at securing the technology they buy, leaving CISOs and their teams playing an increasingly expensive game of whack-a-mole as vulnerabilities keep appearing. AI now turbocharges that cycle. Customers will spend more, cybersecurity companies will grow and software companies will continue to release increasingly powerful technology.

A properly functioning market would put much greater economic pressure on suppliers to reduce the insecurity and unsafe behaviour they introduce in the first place. Cybersecurity will still be a huge industry, but its growth should not depend so heavily on compensating for weaknesses in technology that customers have purchased.

Adam Smith understood 250 years ago that markets are shaped by self-interest and incentives. The cybersecurity market is no exception. Change the incentives and the market will change with them. Ironically, a smaller cybersecurity market would be a sign of success and, in Smith’s terms, a more rational market.