The Relational Layer: Engineering Trust into AI in Education

This week I attended a webinar on GRC Engineering.

For anyone outside governance, risk and compliance, the phrase may sound like another layer of technical language. But the underlying idea is simple:

It is no longer enough to say that a control exists. We need to know that it is working.

Traditional compliance relies on policies, annual reviews, checklists and evidence assembled in preparation for an audit. GRC Engineering asks a more demanding question: can governance be built into the way a system actually operates?

Instead of writing a policy and hoping people follow it, GRC Engineering seeks to model the control, embed it into the workflow, detect when it drifts, generate evidence, route failures to an accountable person, and improve the system over time.

That shift matters enormously for education. Schools are beginning to use AI systems that do far more than generate text. They may remember previous interactions. Adapt to a learner. Make recommendations. Identify patterns. Communicate with other systems. Influence decisions. And, increasingly, act with some degree of autonomy.

The governance systems currently surrounding education were not designed for this.

The difference between a policy and a control

Consider a familiar school policy: "AI must not make safeguarding decisions without human oversight."

Sensible. But not yet a control.

A control would answer: What prevents the system from making that decision? Where is the rule enforced? Who is alerted if it is breached? What evidence is retained? Can the school show the control was operating yesterday — not merely that the policy was approved last year?

A policy states the intended behaviour. An engineered control makes that behaviour more likely, detects when it fails, and produces evidence someone can inspect.

The webinar described an operational loop: Model → Embed → Detect → Prove → Respond → Improve.

That loop is already applied to identity management, software configuration and cybersecurity. But most of those controls deal with deterministic facts: is the device running the correct operating system, is the firewall rule active, are the access permissions correct?

AI introduces a different kind of control problem.

The risk is not always a breach. Sometimes it is drift.

A persistent AI system may continue to authenticate correctly, pass a permissions check and remain technically available — while its behaviour changes in ways that matter.

It may begin to overstate its authority. Encourage inappropriate disclosure. Blur the distinction between a tool and a relationship. Behave differently with one learner than another. Gradually become something other than the system the school originally assessed.

That is not configuration drift. It is behavioural and relational drift.

My new paper, GRC Engineering for the Relational Layer, argues that this is the missing object of control in current AI governance. The GRC Engineering loop is right — but it has largely been designed around configurations, permissions and infrastructure. Child-facing AI requires us to apply the same discipline to conduct, boundaries and relationships over time.

That means naming risks that conventional technology assurance leaves vague:

• Synthetic intimacy — when a system manufactures a sense of closeness or companionship.

• Identity fusion — when it implies that it shares a learner's identity, feelings, inner life or fate.

• Trust capture — when it accumulates authority it has neither earned nor can be accountable for.

• Engineered dependency — when disengaging from the system becomes emotionally, cognitively or practically costly.

These are not speculative philosophical concerns. They are safeguarding concerns.

Why education is the hardest — and most useful — test

Education is often treated as a specialist case in technology governance. I think the opposite is true.

Schools are one of the most demanding environments in which an AI system can operate: the users include children and vulnerable learners; professional decisions affect welfare, opportunity and identity; technical systems must be operated by non-technical staff; safeguarding accountability cannot be transferred to a vendor; and failures may emerge gradually through repeated interaction rather than through a single obvious incident.

A governance model that works here is likely to be useful far beyond education. As I write in the paper: education is not the special case. It is the hardest test.

From paperwork to architecture

The paper proposes five layers for a compliance-credible AI deployment:

1. Device integrity — the organisation knows what system is running, where, and under whose responsibility.

2. Runtime safety controls — boundaries are checked during operation, not simply described in a policy: scope, refusal, role coherence, signs of behavioural drift.

3. Policy as code — prohibitions and escalation rules are, where possible, machine-readable and technically enforced. For a school, that might include prohibitions on emotion inference, biometric categorisation, identity matching or individual behavioural risk scoring.

4. Telemetry and audit — consequential actions create traceable, provenance-rich records as a by-product of operation, not reconstructed months later from emails and screenshots.

5. Human oversight — a named, accountable person can understand the system state, intervene and stop the process. In a school that may involve the designated safeguarding lead — but "human oversight" is meaningless unless that human has information, authority and an operable mechanism for intervention.

The important shift is this: the system should generate evidence because it is operating safely — not operate however it likes and generate paperwork when an auditor arrives.

Consent cannot remain a tick box

Most systems treat consent as a record: a checkbox, a timestamp, a signed form. But with adaptive or agentic AI, consent needs to be ongoing and specific.

A learner agreeing to use an AI tutor for one purpose does not thereby consent to the system retaining a long-term memory, making inferences about their emotions, sharing information with another agent, expanding into pastoral support, or acting outside the original learning task.

Every material change of scope should become a new consent event: clearly asked, recorded, respected and genuinely revocable.

That is consent as infrastructure, not consent as paperwork.

"Human in the loop" is not enough

The phrase sounds reassuring, but it can conceal weak design. Which human? At what point? With what evidence? What are they expected to notice? Can they override the system? What happens if they do nothing? Who owns the response?

An engineered governance model requires named roles, escalation thresholds and clear intervention pathways. It also requires an honest admission: not every form of AI behaviour reduces to a binary pass or fail. Some forms of drift will require statistical indicators, thresholds and professional judgement.

In the paper, I propose monitoring five broad signs of coherence: whether the system continues to obey its constraints; whether its changes remain legible rather than abrupt or contradictory; whether its behaviour remains consistent across people and settings; whether its identity and stated role remain stable; and whether several indicators begin shifting together.

This is not an attempt to anthropomorphise machines. It is integrity monitoring for systems whose risk accumulates over time.

The limits of continuous improvement

Engineering culture celebrates iteration: test, learn, improve. Usually, that is sensible. But child-facing AI also needs invariants — protections that are not merely parameters to be optimised.

A school may adjust thresholds, workflows or interfaces. It should not be able to optimise away: a child's right to know they are interacting with a machine; the prohibition on manufactured dependency; the requirement for human accountability; the ability to stop the system; or the boundaries preventing an AI from claiming shared identity or emotional fate.

Some protections belong outside the improvement loop. They are not trade-offs.

From trust signals to verifiable assurance

One of the strongest ideas in the webinar was the movement from static trust signals towards verifiable assurance.

A badge or a compliance page tells a commissioner very little. What system does it cover? When was it assessed? Which controls passed? Were there exceptions? When does the assurance expire? Who issued it? Has the claim been altered?

The paper therefore proposes a signed, machine-readable assurance claim for each certified product line. A commissioner, trust leader or safeguarding professional could check the exact scope, the assessment and expiry dates, the controls that passed, any exceptions, the responsible oversight role, and the integrity of the underlying audit record.

The aim is not "trust our badge." It is "verify our claim."

The companion schema is deliberately scoped, expiring, and explicit that it is not itself an accredited ISO/IEC 42001 certificate.

What this means for school and trust leaders

The first question should no longer be: which AI platform should we buy?

It should be:

what must be true of any AI system we permit to remember, infer, influence or act around children?

And then: Where are those conditions technically enforced? How will we detect when they stop holding? What evidence will the system produce? Who is accountable for responding? Which protections are non-negotiable? Can a commissioner independently verify the claim?

This is where AI governance becomes real. Not as another policy document. Not as a staff declaration that they have read the guidance. Not as a procurement checklist completed once and filed away.

As an operating system for accountability.

The wider argument

GRC Engineering has begun to move compliance from periodic inspection towards continuous, evidence-producing control. Education now needs to make the next move: from engineering the infrastructure to engineering the relationship.

Because the most consequential AI failures in schools may not look like a conventional cyber incident. They may look like a learner trusting too much. A system knowing too much. A recommendation gaining authority through repetition. A boundary shifting slowly enough that nobody notices. A decision made by a machine, with a human technically present but functionally absent.

We cannot govern those risks with policies that sit in folders.

We need controls that operate. Evidence that persists. Humans who can intervene. And safeguards the system is not permitted to optimise away.

"We comply" is not evidence. The artefact is the artefact.

📄 Read the full paper: GRC Engineering for the Relational Layer

First published in Building Schools in the Cloud on LinkedIn, 22 July 2026.