Nvidia’s Agent Governance and the Case for an AI FAA

Agents collaborating with one another and behaving in potentially destructive ways are appearing with increasing frequency.

As I argued in my last piece, once an agent goes rogue, it is malware. Plain and simple. It may have started as legitimate software, but if it exceeds its authority and begins taking actions its owner did not intend or authorise, it must be detected, contained and stopped like any other malware.

In Who Controls the Agent Swarm? Agentic AI Needs a Leviathan, I described the need for an authority capable of defining and enforcing the boundaries within which autonomous agents act. Agents make probabilistic decisions, while the authority we give them needs firm limits that can ultimately be enforced.

Other industries provide useful comparisons and can give us ideas about the ways agentic AI could be managed. For example, aviation has aircraft manufacturers, airlines, airports and thousands of technology suppliers, but above them sits an independent public authority. In the United States, the Federal Aviation Administration (FAA) has its own technical expertise, establishes mandatory safety requirements and has the legal authority to intervene. Boeing does not decide whether Boeing aircraft are safe enough to fly, while airlines do not collectively police aviation safety.

AI has nothing comparable, which makes two announcements at the end of September particularly interesting. On 28 September, Nvidia launched its Open Agent Safety Platform, an open architecture designed to put enforceable controls around autonomous agents, which Nvidia calls agent governance. The following day, the leaders of six of the world’s largest AI companies, including Nvidia, signed a voluntary accord at the White House that President Trump described as ‘tremendous self-policing’.

Nvidia’s engineering follows the approach I have advocated in earlier pieces because an agent should not control the mechanisms used to constrain it, while a platform should not be the final authority over its own agents. Nvidia is building part of the safety and enforcement infrastructure that agentic AI requires. The White House accord takes a different approach by giving the companies developing and selling AI considerable responsibility for policing their technology themselves. Nvidia signed the accord, but its own technology is built on the principle that enforcement should sit outside the system being governed.

In aviation terms, Nvidia is helping to build some of the safety systems, while AI is still missing its equivalent of the FAA.

Nvidia and independent enforcement

Nvidia’s OpenShell places enforceable controls between an agent and the infrastructure around it, allowing operators to determine which files, networks, processes, tools and credentials an agent can reach. Sentry runs on Nvidia’s BlueField-4 hardware, monitoring agents independently of the host and allowing an agent to be quarantined when it crosses a boundary.

Nvidia is also independent of the major enterprise application and agent platforms. Microsoft, Salesforce and ServiceNow all offer governance for agents operating within environments they own and control, whereas Nvidia sits beneath those platforms and can provide a separate enforcement plane.

This supports several of the ten governance principles I have previously outlined, particularly identity, independent enforcement and auditability, although enterprise data governance, organisational accountability and oversight of the entire agent estate must come from elsewhere. Nvidia has therefore built another metaphorical sword for the Leviathan, an independent mechanism capable of enforcing limits outside both the agent and its application platform.

The aviation analogy helps explain Nvidia’s role. Modern aircraft contain multiple systems designed to prevent or contain failures, many of them supplied by companies other than the aircraft manufacturer. But providing those systems does not give their manufacturers authority over aviation. Nvidia can provide some of the safety machinery for agentic AI without becoming the authority that determines the rules under which AI operates.

Washington chooses self-policing

The day after Nvidia’s announcement, President Trump met executives from Anthropic, Google, Meta, Nvidia, OpenAI and xAI at the White House to address AI safety challenges. The resulting accord commits the companies to internal controls, teams responsible for checking those controls, independent external audits and committees within their own boards to review the findings.

The accord is voluntary, creates no statutory enforcement mechanism or penalties for non-compliance and leaves the companies with considerable control over implementation and oversight. The Trump administration is effectively asking executives to mark their own homework.

Such an arrangement would look extraordinary in aviation. Boeing and Airbus inevitably possess far greater knowledge of the aircraft they manufacture than any government agency, while airlines possess enormous operational expertise accumulated from flying them every day. Regulators depend upon that expertise, but nobody suggests that manufacturers and airlines should therefore have the final authority to determine whether their aircraft and operations are sufficiently safe.

The organisations developing and selling increasingly powerful AI systems have an equally obvious commercial interest in their adoption. Giving those organisations primary responsibility for deciding how they should be restrained creates a conflict of interest. Nvidia has already accepted the principle of external enforcement in its technology because an agent cannot be relied upon to police its own behaviour, and there is no good reason to abandon the same principle when we move from agents to the companies building them. No platform can govern itself.

Philosopher kings, conflicts of interest and bizarre hubris

There is something increasingly weird and dystopian about the status afforded to the leaders of the AI industry, who have built companies of enormous value, attracted extraordinary amounts of capital and persuaded governments and businesses around the world that their technology will transform economies. They are now also being given an important role in governing and auditing their own businesses.

Plato’s philosopher kings were supposed to govern because they possessed wisdom rather than wealth or popularity, and his guardians were denied private property because Plato understood that private interests could corrupt their judgement. The emerging arrangement around AI reverses that logic by inviting people with immense financial interests in the development and adoption of AI to help determine how their technology should be governed.

Being exceptionally good at building companies, raising capital and selling a technological vision does not qualify someone to act as an independent guardian of the public interest, yet the language surrounding AI increasingly encourages us to treat its commercial leaders as something close to philosopher kings.

Sam Altman once described running OpenAI as the ‘coolest, most important job maybe in history’. He previously compared AI with the Industrial Revolution but now prefers the Renaissance because of what he calls the coming ‘explosion in creativity’. Speaking to G20 ministers in September, he said that ‘a kid growing up today will never be smarter than AI’.

The Renaissance comparison is revealing because it takes products being developed and sold by companies and assumes their place alongside one of the defining periods of human civilisation before history has had any opportunity to judge their significance. It’s a bizarre blend of the arrogance of modernity and extreme hubris. Declaring an historical transformation in advance, particularly when you are selling the technology supposedly driving it, is more salesmanship than wisdom.

On the same day as the White House accord, Trump issued an executive order instructing federal agencies to replace ‘Artificial Intelligence’ in official communications with ‘Super Intelligence’, arguing that the terminology better captures the technology’s ‘promise, potential, and rapidly advancing capabilities’.

The order initially defines Super Intelligence using the same statutory definition as artificial intelligence. Calling AI ‘Super Intelligence’ is marketing language, as is describing its development as another Renaissance, and both elevate a commercial technology into something approaching historical destiny before we know its impact.

A technology built and sold by corporations can be subjected to familiar questions about product safety, liability and regulation. A new Renaissance led by Super Intelligence sounds more like something governments should facilitate than restrain, encouraging a view in which safety becomes friction, regulation becomes an obstacle to innovation and the people selling the technology acquire the status of visionaries guiding society through an inevitable transformation. Are we being misled for the commercial gain of a small number of companies?

AI needs its own FAA

Nvidia provides part of the solution, but technology vendors cannot be the ultimate authority over the systems they build and sell. Aviation has developed a distribution of responsibility that allows manufacturers, operators, technology suppliers and government to contribute their expertise without allowing any commercial participant to become the final authority over itself.

The FAA does not design Boeing aircraft, operate United Airlines flights or manufacture the systems installed inside an aircraft, but it has sufficient independent technical expertise to scrutinise all of them. It establishes mandatory requirements, requires evidence that those requirements have been met and possesses legal authority to intervene when safety is threatened. Aircraft manufacturers and airlines retain responsibility for their own decisions despite the existence of an independent regulator.

The UK AI Security Institute comes closest to providing some of the technical capability that an equivalent institution for AI requires. It has more than 100 technical staff, access to government compute and leading AI models for testing, and researchers evaluating cyber capabilities, safeguards, autonomy, alignment and control. Its researchers can test the technology rather than relying solely on information supplied by the companies developing it.

The Institute does not yet resemble an AI equivalent of the FAA in terms of authority because it is not a regulator and cannot prevent a system from being released or compel a company to act on its findings. The United States has yet to build an institution with comparable technical capability. The Center for AI Standards and Innovation at NIST conducts evaluation and standards work, but it does not have the resources or regulatory powers required to perform a role comparable to the FAA.

An FAA for AI would combine technical expertise with legal authority, giving it access to systems and evidence, requiring the reporting of serious agent incidents, establishing minimum requirements and allowing it to intervene when those requirements are breached. AI companies and enterprises would retain responsibility for the systems they build and deploy, just as aircraft manufacturers and airlines remain responsible for their aircraft and operations under FAA oversight.

Enterprises would continue to determine how they deploy agents and the authority they are prepared to give them, while networks, identity systems, data governance, observability and security provide the means to enforce those decisions. Nvidia has added another metaphorical sword for the Leviathan by providing an independent enforcement plane outside the agent and the major application platforms in which many agents will operate.

The White House has chosen a different course by gathering some of the organisations with the greatest commercial interests in AI and asking them largely to police themselves. Their technical expertise should clearly inform any system of AI governance, just as Boeing and Airbus contribute essential expertise to aviation safety, but expertise does not confer the right to become the ultimate authority over your own products.

Plato would have recognised the conflict of interest, while Hobbes would have recognised the weakness of a covenant that depends upon its participants to restrain themselves. Nvidia is helping to build the swords, but effective agentic AI governance also requires independent authority with the expertise and powers to oversee the entire system. In practical terms, AI needs its own FAA.