Trust fails first.

Over the past year, many organisations have moved from AI experimentation to production. Expectations around accountability, transparency, and regulatory scrutiny have hardened in step. What I see repeatedly is not a lack of capability, but a gap between what AI systems can do and what organisations are actually prepared to stand behind when those systems influence customers, payments, decisions, or outcomes.

In regulated and high-trust environments - payments, identity, financial services, public sector - AI cannot be treated as a feature or a bolt-on. It has to sit inside a coherent operating model: clear data boundaries, explainability, control points, and decision ownership that executives are willing to defend. Architecture, standards, and governance are not brakes on innovation. They are what allow innovation to scale without eroding trust.

The expanded thinking below draws on the exchange that followed the original post. Several of the framings are not mine - they came from contributors whose comments sharpened the argument enough that it deserved to be set down properly. Where their framings have shaped the thinking, I have named them.

The shift the past year has surfaced

A year ago, most enterprise AI conversations centred on capability: which model, which vendor, which proof of concept. Twelve months later, the question has changed. Capability is broadly established. The harder question now sits in front of every leader who has to put their name to a decision: when this system acts, are we prepared to defend what it did?

The organisations that get this right do not get there by adding more controls on top of an existing process. They redesign the operating model so that ownership, escalation, and challenge are explicit before the AI is allowed to act. That work is unglamorous. It also separates the programmes that scale from the ones that quietly stall the moment scrutiny arrives.

Trust as outcome, not assumption

Stephan Engberg made a sharp challenge to the framing of trust itself: “the absence of trust is inherent so designing to isolation and contain risk is critical”. The point holds.

It is fair to ask whether “trust” is the right framing. In high-risk systems, the absence of trust is the correct starting point. Containment, isolation, and explicit accountability have to be designed in from day one. Trust is not a precondition; it is a property that emerges, or fails to emerge, from how a system is built and governed.

That distinction matters because it changes where leaders look for the failure. When AI-driven decisions go wrong, the immediate instinct is to interrogate the model. Ahmed AlSaidalani made the same observation in plainer language: most AI ‘failures’ we see today are not algorithmic, they’re organisational. AI is rarely the most useful place to look. The more revealing question is whether the organisation had a coherent decision model in the first place, and whether AI made the cracks in that model visible at machine speed.

Ashby’s Law, applied to leadership

Daniel Jonas, a global payments strategist and practising cybernetician, made the cybernetic point in response to the original post. Ashby’s Law of Requisite Variety states that the regulating body of a system must be able to cope with the variety of the system being regulated.

AI substantially increases the variety of decisions an organisation produces and the speed at which those decisions land. If governance, ownership, and escalation do not increase in sophistication and responsiveness in step with that, the regulating body - whether that is an internal control function, a board committee, or an external authority - is no longer matched to the system it is meant to oversee.

This is the cybernetic version of the same observation many practitioners are making in plainer language: governance designed for human-speed decisions does not hold at machine speed. The work, then, is not to slow AI down to the cadence of yesterday’s controls. It is to redesign the controls so that they remain meaningful at the cadence AI now moves at.

Authority is what trust is built on

A pattern runs through the most thoughtful responses to this argument. Wong Jia Qing put it as cleanly as anyone: trust usually fails downstream of a simpler problem, unclear decision authority. Trust failures, examined carefully, are usually authority failures. Was the action authorised, by whom, within what scope, for what duration, with what conditions on revocation? When the answer to those questions is “we would have to reconstruct it from the logs”, the system was already indefensible before anything went wrong.

Design-time policy is necessary but not sufficient. Authority has to be expressible, enforceable, and observable at the moment an action is taken. Susan Brown captured this directly: static credentials and standing permissions break down once execution shifts to agents and machine-speed workflows. They become liabilities when the actor in question is an autonomous agent committing actions in milliseconds.

Execution as evidence

Dean Chapman pressed the argument further. Explainability addresses reasoning. Reality addresses what actually happened. As Dean put it, “the real trust gap isn’t ‘Who decided?’, it’s ‘Did it actually happen as recorded?’.

There is a further step beyond explainability. Explainability addresses reasoning. Reality addresses what actually happened. In high-stakes systems, accountability requires more than a defensible explanation of why a decision was made; it requires verifiable proof that the decision led to the action recorded, on the system claimed, under the authority asserted, at the moment stated.

Without that, “auditable” becomes a narrative rather than a fact. The organisations preparing for this most rigorously are the ones investing in execution-level evidence as a first-class concern, alongside the model and the policy. That is a different conversation from “is the model accurate?”, and a more useful one for any leader who will eventually have to stand in front of a regulator.

The next twelve to twenty-four months

The organisations that get this right over the next twelve to twenty-four months will not be the fastest adopters. They will be the ones whose AI decisions still make sense, and still stand up, when attention turns from possibility to accountability. That turn is not a question of if; it is a question of when.

In the meantime, the most useful question a leader can put to their own organisation is the one a regulator will eventually put to them: if an AI-influenced decision causes harm tomorrow, who is genuinely empowered to stop the system, explain the outcome, and stand behind it publicly? The quality of that answer, or the absence of one, is where trust really lives.

AI does not fail first. Trust does.