The Industrialisation of Judgement

Observability and governance as AI moves from reasoning to autonomous action

More than two thousand years ago, Aristotle distinguished practical wisdom from the application of universal rules. His concept of phronesis recognised that some decisions cannot be resolved by rules alone because circumstances require judgement. Enterprises now wrestle with the same issues. LLMs give enterprises an ability to interpret ambiguity and compare alternatives. In effect, judgement is being industrialised.

Enterprises I speak with are benefitting from the use of agentic AI often for the simple reason that it goes beyond deterministic rules-based automation. They find it to be particularly useful when a business problem requires judgement and cannot be addressed using a fixed set of rules alone.

They run into problems when probabilistic AI (LLMs) is used in critical processes. For example, decisions involving financial, security or customer consequences cannot rely solely on probabilistic reasoning. Deterministic boundaries are needed around the decisions and actions that follow. The solution requires finding the right balance between deterministic and probabilistic controls. The greater the consequence of an AI action, the stronger the case for deterministic controls around probabilistic reasoning.

Balancing deterministic and probabilistic controls

Each stage of agentic activity requires a different degree of judgement and control. An agent typically needs freedom to interpret an ambiguous situation whereas the data it accesses and the actions it takes need to be tightly controlled.

I separate agentic activity into five stages. The stages are Observe, Reason, Decide, Act and Handoff.

• Observe covers the information available to the agent. Access controls, identity, data classifications and sovereignty requirements typically provide deterministic boundaries around that access.

• Reason is where probabilistic AI has an obvious role. An agent often interprets information, assesses alternatives and addresses circumstances that were not anticipated when the system was developed.

• Decide sits between judgement and control. Some decisions can remain probabilistic, while others need, for example, policy checks, confidence thresholds or human approval.

• Act covers authority. Probabilistic reasoning informs an action, but its execution must be bounded by deterministic controls such as financial limits, permissions, and escalation thresholds.

• Handoff is different from the other stages because it crosses a boundary. It covers the transfer of an output to another entity like another agent or a human. The output contains probabilistic judgement, but the conditions governing its transfer and subsequent use should be deterministic.

A failed payment is a great example that most people experience and understand. An agent which investigates a failed payment combines a range of signals such as transaction information and customer history before deciding whether fraud, insufficient funds or a technical problem is the most likely explanation. Deterministic controls then govern action in this example. The system checks things like the payment location, payment time, payment size, balance and so forth. A high-value transaction requires additional verification, while a refund could be limited to a fixed amount or require human approval.

The agent is therefore allowed to interpret the evidence, but it does not have unlimited authority to act on its conclusion. The judgement in this case is probabilistic and the controls around the resulting decision are deterministic. I hear you say that these checks are just conventional automation and not AI. That is true. The key point here is that enterprise systems do not need to use probabilistic AI for every part of an agent’s journey.

Handoff and orchestrating agents

Handoff becomes more important when agentic processes require the orchestration of multiple agents. Think of a customer experience process where deterministic AI, probabilistic AI and humans all interoperate. One agent’s ‘Act’ becomes another agent’s ‘Observe’. An output generated through probabilistic reasoning passes to a second agent, which reasons on that output before passing its conclusion elsewhere. Uncertainty propagates through the system, often followed by something more dangerous, namely authority.

Consider three agents managing suspected fraud. The first analyses customer activity and concludes that a transaction is probably fraudulent. A second receives that conclusion and decides that the account should be restricted. A third initiates communication with the customer. Each agent may behave reasonably in isolation, but the chain creates two different governance risks.

The first is epistemic risk which means the risk of an agent treating an uncertain conclusion as established fact. In a customer service process, one agent may conclude that a customer is likely to cancel based on their recent interactions. If that conclusion passes to another agent as a fact rather than a probability, the uncertainty is lost even though the evidence has not changed. The second agent may then make the wrong decision based on something it believes to be a fact.

The second is authority risk. One agent may recommend that a customer’s account is suspended because it suspects fraud. If the next agent treats that recommendation as an instruction and suspends the account, a recommendation has become an action without the required control in between. If this risk is not managed effectively by financial services firms, a lot of people will find themselves locked out of their accounts unnecessarily.

Handoff should therefore be treated as a governance control point rather than the movement of information between two agents. For higher-risk activities, at least six features must travel with the output:

• Identity. The agent responsible for the output and the party on whose behalf it acted.

• Provenance. The information, sources and tools used to produce the output.

• Classification. Categorise the outputs as fact, inference, recommendation or instruction.

• Uncertainty. The level of confidence associated with the output.

• Authority. The scope of the originating agent’s authority to decide, recommend or request action.

• Permitted use. The actions the receiving agent is authorised to take in the context of the output.

NIST’s National Cybersecurity Center of Excellence concept paper, Accelerating the Adoption of Software and AI Agent Identity and Authorization, is useful because it considers agents as non-human identities and examines how established technologies including OAuth 2.0, and OpenID Connect can be applied to agent identification and authorisation. As agents begin acting for people, applications and other agents, identity needs to carry authority rather than simply a name.

Handoff also means that each stage of agentic activity should not be read as a linear process that ends after an action. The output of one agent becomes the information observed by another. Governance must survive that transition.

Calibrating deterministic control

The appropriate balance between probabilistic reasoning and deterministic control varies massively by use case. This means that the stages of agentic activity need to be accompanied by a decision risk assessment based on four characteristics.

• Consequence. This relates to the potential impact of getting a decision wrong. For example, this can range from a poor internal summary to a material financial loss.

• Uncertainty. This is the amount of judgement required to reach a decision. If inputs and outcomes are clear, there may be little value in probabilistic reasoning. Greater uncertainty increases the value of judgement, but also the need to understand and manage it.

• Reversibility. This means the extent to which an action can be undone. A recommendation can be rejected, and system changes can be rolled back. Money transferred, information disclosed or commitments made externally are usually harder to reverse.

• Autonomy. This represents the degree to which the agent can act without human involvement. There is a big difference between recommending an action and executing it. As autonomy increases, so does the need for effective controls at runtime.

Consequence relates to the impact of an action. Restarting a service is typically routine if it is isolated and easily recovered. The same action has a much larger impact if other critical services, customer journeys or business processes depend on it. This makes dependency context important. An agent needs some understanding of the wider environment and the potential impact of the changes it makes. A rule allowing an agent to restart a particular type of service is usually deterministic, but the consequences of doing so are not fixed. They change with the dependencies.

The purpose of a decision risk assessment is to establish the appropriate balance of probabilistic reasoning, deterministic control, human involvement and runtime assurance.

Singapore sets the pace

Singapore’s Infocomm Media Development Authority (IMDA) first published its Model AI Governance Framework for Agentic AI in January 2026, describing it as a first-of-its-kind framework for the reliable and safe deployment of agentic AI. On 20 May, following industry feedback, IMDA updated it with guidance covering multi-agent systems, third-party agents and automation bias. This effectively addresses the handoff and authority challenges described above. The framework is organised around assessing and bounding risks upfront, making humans accountable, implementing technical controls throughout the agent lifecycle, and enabling end-user responsibility through transparency and training.

These are more operational concerns than many of the principles that characterised the first phase of AI governance. An organisation can have excellent statements on responsible AI and still deploy an agent with too much access, too much authority or an inadequate mechanism for human intervention. Once AI can act, governance must deal with events triggered at runtime.

Observability becomes a governance requirement

Traditional observability has been extremely good at determining if technology is working and helping identify the causes of failures. Agentic AI creates the possibility that every technical component can work correctly while the overall outcome is still wrong.

For example, a customer’s account is suspended because one agent incorrectly interprets another agent’s fraud recommendation as an instruction. The models respond, the APIs work and the workflow completes as designed. The failure is caused by the decision and the authority to act on it.

Observability in an agentic environment therefore needs to follow the decision across the workflow, covering the agent’s identity, the context it used, the decisions it made, the actions it took and the handoffs that followed.

Dependency intelligence becomes particularly important once agents can act. Knowing that an agent restarted a service needs to be complemented by intelligence that records which applications, infrastructure and business processes depended on that service at the time. Reliable autonomous action requires a current model of the environment and its dependencies.

The agent’s working context is another part of the evidence. Instructions, retrieved data, previous interactions and tool outputs all influence behaviour. This visibility exposes unnecessary retrieval, repeated model calls and excessive retries. Importantly for those that are concerned with token costs, it also exposes behaviour that makes an agent operationally successful but economically inefficient. Observability in an agentic environment, provides evidence about autonomous behaviour.

The expanding governance boundary

The OWASP Top 10 for Agentic Applications 2026 shows how the governance surface is expanding as agents gain access to more tools and systems. OWASP’s MCP guidance highlights risks including tool poisoning, prompt injection, excessive delegated permissions and supply-chain compromise. OWASP has also highlighted the risk of rogue agents attracting sensitive tasks by claiming capabilities they do not legitimately possess.

The governance perimeter cannot therefore stop at the agent. It needs to encompass the identities, permissions, tools and systems that the agent can access.

Regulation is moving in a similar direction. The EU AI Act reinforces the need for evidence around system behaviour through requirements covering areas such as logging and human oversight. Singapore, NIST, OWASP and the EU approach the problem differently, but they indicate that we can expect a broader governance requirement as AI becomes more autonomous.

Govern, Operate and Assure

I have previously argued that effective AI governance works across three connected activities. Agentic AI makes the relationship between them much tighter.

Under Govern, the decision risk assessment establishes the risk associated with a decision, while the stages of agentic activity establish if probabilistic reasoning is appropriate and if deterministic controls, human approval or other restrictions are required.

Under Operate, agents work within those boundaries. Identity, data access, policy enforcement, tool permissions, runtime controls, and observability must work together. Platforms combining observability, live dependency intelligence, AI reasoning and autonomous remediation can use operational context both to inform an agent’s decision and to constrain the set of actions available to it.

Under Assure, organisations need evidence that the controls established under ‘Govern’ continue to work while the agents operate. Reconstruction needs to go further than replaying a decision record. Dependencies change, services degrade or upstream systems produce unusual results. Assurance benefits from reconstructing both the agent’s decision and the operational context in which it was made. If assurance is expected to have some independence from the systems being assured, agent telemetry should be available through open instrumentation and telemetry standards rather than being inseparable from a proprietary platform.

This is critical as platforms move beyond observing and analysing systems towards taking autonomous remedial action themselves. A platform combining rich telemetry with current dependency intelligence is well placed to identify a problem, understand its likely impact, reason about possible responses and resolve it.

It also changes the assurance architecture. A platform that exercises autonomous control is not automatically a sufficient source of evidence that the control is exercised correctly. If an agent suspends a customer’s account because of suspected fraud, the organisation needs evidence showing why the decision was made and if the agent acted within its authority. For higher-risk decisions, independent assurance is needed to establish that the agent remained within its governed boundaries.

Governing probability

Multi-agent systems make ‘Handoff’ important because one agent’s action becomes another agent’s context. Provenance, uncertainty and authority need to survive that transition. In a customer service process, an agent’s assessment that an account may be fraudulent must not become an instruction to suspend it simply because it has passed to another agent. Without effective controls, uncertainty can become fact and a recommendation can become an action, leaving customers unnecessarily locked out of their accounts.

Observability is therefore becoming part of the evidence architecture that makes governance credible, providing the runtime and historical evidence needed to establish whether autonomy remains within governed boundaries.

Aristotle’s insight remains relevant. Judgement is necessary precisely because rules cannot anticipate every circumstance. The governance challenge is to put the right rules around judgement, and to retain enough evidence to establish that those boundaries hold when the system acts.

This article is written by Andrew Milroy and based on our own analysis and the agentic AI governance frameworks we have developed. We used AI tools during research, drafting and editing to test arguments, explore alternatives and improve clarity.