Automated decisions, human consequences: the AI governance imperative for insurance distribution and what it means for your business

Distributing insurance can involve a range of regulated activities requiring FCA authorisation, including advising, arranging, and introducing, but these concepts were designed with human intermediaries in mind. Where an AI system facilitates the conclusion of an insurance contract, whether it is "arranging" that contract requires careful analysis of both the customer journey and the specific functionality deployed. The FCA has acknowledged this ambiguity but has yet to produce definitive guidance specific to AI-generated insurance recommendations, leaving firms to self-classify and creating a risk of regulatory arbitrage where technologically advanced distribution models operate with lighter-touch oversight than their human-led equivalents performing the same function.

We have seen this issue firsthand when assisting clients with developing AI distribution models and advising on what distribution model a firm can implement without exposing it to unnecessary risks of activities being carried out where the correct permissions are not held.

The FCA's frontier AI review reinforces this concern. The FCA's engagement with firms signals clear supervisory expectations, and firms, particularly small to medium-sized firms, should treat the findings as an early indicator of where regulatory focus will settle.  Firms operating AI-assisted distribution models who have not yet reviewed their regulatory classification should treat the frontier AI review as a prompt to act now, before supervisory focus intensifies.

AI-powered tools are increasingly being used to deliver product recommendations to consumers with minimal or no human involvement, ingesting customer data such as financial position, risk appetite, and lifestyle factors to generate tailored recommendations. Yet many technology-driven distributors continue to characterise their systems as providing "information" rather than "advice", a distinction that is becoming increasingly difficult to defend as AI systems become more sophisticated and personalised. Firms should urgently review how their systems are classified, given that the advice label triggers significantly higher regulatory obligations including suitability assessments, conflicts of interest management, and enhanced disclosure requirements.

The frontier AI review adds an important operational dimension here. Firms using frontier AI models report that the value they get from them is determined less by the models themselves and more by the technical and operational environment in which they are deployed, including specialist tooling, robust validation processes, operational guardrails (such as limits on model permissions, human approval for higher-risk actions, and controls over access to sensitive systems and data), and human expertise.  The same logic applies in distribution: it is not the sophistication of the recommendation engine alone that determines regulatory exposure, it is the governance framework, validation controls, and human oversight architecture that surrounds it. A system characterised as providing "information" will struggle to sustain that classification if it is operating with AI-generated personalised outputs but without any of the human oversight infrastructure the FCA now clearly expects.

Firms must be able to demonstrate that their automated recommendation engines genuinely serve consumer interests rather than optimising for commercial metrics such as conversion rates or premium income. Where an algorithm has been trained on historical data that reflects prior mis-selling patterns or demographic biases, the resulting recommendations may systematically steer consumers towards unsuitable products, in breach of Consumer Duty obligations. The consumer understanding outcome is also directly engaged where AI-generated communications, for example chatbot outputs, are used in the distribution process, and firms must demonstrate that those communications are accurate, clear, not misleading, and enable customers to make informed decisions.

The frontier AI review's finding on vulnerability discovery provides a direct operational analogy for insurance firms. Frontier AI is increasingly supporting the identification, validation and prioritisation of vulnerabilities, increasing the pressure on firms' remediation processes. Even where a substantial proportion of model outputs are ultimately discounted through expert review, the remaining volume of genuine vulnerabilities can still put considerable pressure on remediation teams, engineering resources, and change management processes.

Insurance firms deploying AI in distribution face an analogous challenge: as AI recommendation engines and claims-assessment models generate outputs at scale, the volume of decisions requiring expert review before deployment or approval can rapidly outpace the firm's human validation capacity. A model that generates hundreds of product recommendations per hour, or triages thousands of claims per day, creates an asymmetry between what the system can produce and what compliance, actuarial, and conduct risk teams can meaningfully assess. Firms should map this validation bottleneck explicitly before scaling AI distribution tools and should not assume that post-deployment monitoring is a sufficient substitute for pre-deployment validation at scale.

The Consumer Duty does not distinguish between human and automated decision-making, a firm cannot escape liability for poor consumer outcomes simply because those outcomes were produced by a machine, and boards and senior managers bear personal accountability under SM&CR for ensuring that automated systems are appropriately governed and monitored. The FCA has indicated in supervisory communications that it expects a clearly identified Senior Manager to hold accountability for AI systems, meaning that algorithmic failures in distribution can, in principle, be attributed to a named individual with personal regulatory accountability.

The FCA's frontier AI review sharpens this accountability point considerably, moving from principle to practice. Although frontier AI is increasing the degree of automation in cyber resilience activities, firms report that human oversight remains critical, with specialist expertise relied upon to validate findings, assess relevance, determine priorities and make risk-based decisions. Governance forums, risk committees and senior leaders may need clearer visibility of how frontier AI affects vulnerability registers, remediation, supplier dependencies, risk and operational resilience.

Translating this directly into the insurance SM&CR context: it is not enough for a Senior Manager to be nominally accountable for an AI distribution system. The FCA's review makes clear that senior leaders must have meaningful visibility of what the AI is doing:

• how outputs are being validated;
• what escalation routes exist for anomalous results; and
• whether the governance infrastructure genuinely enables human intervention before consumer harm occurs.

A governance framework that logs AI decisions after the fact, without enabling real-time challenge by qualified personnel, will not satisfy this standard.

Firms should ask themselves the following questions in relation to their SM&CR accountability framework for AI:

• Decision ownership: Who, by name and SMF function, owns the decision to deploy each AI system used in distribution or claims? Is this documented in a Statement of Responsibilities or a written governance record?
• Escalation routes: Is there a route to escalate important findings, and are model outputs being assessed by people with the relevant business, conduct and technical experience?
• Validation capacity: Is the volume of AI-generated outputs that require human review proportionate to the capacity of the individuals and teams responsible for reviewing them? Or does the volume mean that human review is, in practice, a rubber stamp?
• Governance visibility: Do governance forums (risk committees, audit committees, board-level discussions) receive sufficiently granular and timely information about AI system performance to enable meaningful oversight rather than retrospective approval?

Many advanced AI models operate as "black boxes", producing outputs that cannot readily be explained even by their developers. This sits uneasily with the requirement that firms provide consumers with comprehensible information to enable informed decisions, and if a distributor cannot explain why a particular product was recommended, it is difficult to see how that obligation is being met. This is well illustrated in fraud detection models, where UK insurers face difficulty explaining to a customer, a court, or the Ombudsman precisely why a claim was flagged, giving rise to litigation risk in cases where claimants allege their legitimate claims were unfairly denied or delayed based on opaque algorithmic assessments.

The frontier AI review reinforces the explainability imperative by identifying what happens when governance does not keep pace with model capability. Frontier AI models can combine multiple lower-rated security flaws (vulnerability chaining) and create alternative routes to compromise. This is encouraging firms to adopt a broader view of risk that looks past severity ratings alone, emphasising a risk-based approach using factors such as exploitability, business service impact, prerequisites to exploit, risk-reduction controls and dependency on the vulnerable system.

The parallel for insurance distribution is direct. Firms must resist the temptation to rely on headline outputs such as a product recommendation, a claims decision, a fraud flag, without understanding the chain of reasoning that produced them. Just as a firm's cyber team cannot responsibly act on a vulnerability output it cannot assess, a firm's distribution or claims function cannot responsibly deploy an AI output it cannot explain or defend. The test is not whether the output appears reasonable on the surface; it is whether a qualified human reviewer can understand, challenge, and if necessary, override the reasoning behind it before the decision affects a consumer.

Frontier AI models are revealing weaknesses not just in technical vulnerabilities, but also in the people, systems and processes that fix them, bringing into sharp focus asset mapping, dependency management, risk ownership, remediation capacity and triage processes. Firms that carry out basic practices well, and have clear accountability and effective oversight, are likely to be better positioned to tackle the challenges frontier AI brings.  For insurance firms, "basic practices" in the AI distribution context means:

• documented model governance;
• regular fairness and accuracy testing;
• output-level explainability requirements in model procurement and development specifications; and
• clear audit trails linking every consumer-facing AI output to an accountable human decision-making process.

Where a machine learning model recommends an inadequate level of cover to a consumer who subsequently suffers an uninsured loss, liability may be contested between the firm that deployed the model, the third-party technology provider that built it, and the data supplier whose training data produced the output. In practice, liability may be contested or delayed or, worse still, borne by the consumer — precisely the outcome the regulatory framework is designed to prevent. Firms should ensure that contracts with third-party AI providers clearly allocate responsibility and that internal governance frameworks can demonstrate meaningful oversight at every stage.

The FCA's frontier AI review places third-party and supply chain risk at the centre of its findings, and does so in terms that directly apply to insurers using externally built or hosted AI models for distribution. Firms highlighted the importance of supplier preparedness, cloud dependencies, software supply chain visibility and shared infrastructure in their environment, as frontier AI may expose risks that require coordinated engagement and information sharing. Some firms are already engaging their suppliers on how they are using AI-enabled vulnerability discovery, how they are validating findings, how they are notifying customers, and whether they are able to remediate quickly.

Insurance firms should apply this supplier engagement discipline to their AI distribution chains. This means going beyond standard technology due diligence questionnaires and actively asking third-party AI providers:

• How are model outputs validated before they are deployed to consumers?
• What testing has been conducted for bias, accuracy, and edge-case behaviour in insurance-specific contexts?
• What notification and remediation processes exist if a model is found to be producing systematically incorrect or unfair outputs at scale?
• Who, within the supplier organisation, holds accountability for model behaviour and how does that interact with the firm's own SM&CR accountability map?

Some firms are not approaching frontier AI as an enterprise-wide capability from the outset, but are instead using targeted deployments as a practical way to test organisational readiness before scaling more widely. This approach is helping firms gain a clearer understanding of where post-discovery validation becomes constrained, how remediation ownership operates, whether change processes can absorb more findings, and how effectively teams can distinguish between theoretical weaknesses and credible exploitable vulnerabilities.

This targeted deployment model is directly applicable to AI in insurance distribution. Firms should resist the impulse to roll out AI recommendation engines, claims triage tools, or automated servicing models at full scale from launch. A phased, segment-limited deployment, for example, piloting an AI recommendation tool with a defined cohort of lower-risk, simpler product journeys before extending to the full product range, allows the firm to stress-test its validation capacity, identify escalation bottlenecks, and build an evidence base for Consumer Duty compliance before the system is operating at a scale where remediation of errors becomes proportionally more difficult and costly.

Aviva, which has been an early adopter of AI across its business, has publicly committed to a set of responsible AI principles governing its use of AI across underwriting, pricing, and distribution, including commitments to explainability, fairness testing, and human oversight of consequential automated decisions, and has established an internal AI ethics review process for new AI deployments. At Lloyd's of London, the Future at Lloyd's programme has included the development of AI governance frameworks applicable to managing agents and coverholders, reflecting a recognition that AI governance requires market-wide standards in a complex, delegated distribution environment. Firms that have not yet developed comparable frameworks should treat these as a benchmark against which their own arrangements will be measured.

The frontier AI review reinforces that governance infrastructure not model capability, is the differentiating factor in regulatory readiness. Firms using frontier AI models report that the value they get from them is determined less by the models themselves and more by the technical and operational environment in which they are deployed, including robust validation processes, operational guardrails, and human expertise.  This observation, drawn from the FCA's direct engagement with regulated firms, maps precisely onto the Aviva and Lloyd's approaches: both prioritise the infrastructure around the AI (human oversight, ethics review, governance frameworks) over the raw capability of the underlying model.

For firms that have not yet reached that standard, the FCA review also offers a practical route: targeted deployments as a practical way to test organisational readiness before scaling more widely.  A firm need not wait until its governance framework is perfect before deploying AI; but it should ensure that every deployment, however limited in scope, is accompanied by a defined validation process, a named accountable individual, and a clear escalation and remediation path. In our experience advising on implementing AI governance structures, firms need a tailored approach to their own business and activities that are utilising AI to ensure a robust and proportionate approach is taken.

AI is increasingly deployed in customer servicing and claims handling, with many insurers and intermediaries now using AI-powered chatbots to handle customer enquiries, guide customers through the purchase journey, assist with claims reporting, and perform claims triage, automatically assessing the validity and complexity of claims and routing them to the appropriate handler. The same regulatory obligations, Consumer Duty, explainability, SM&CR accountability, apply at every stage of the customer lifecycle, not just at the point of sale.

The frontier AI review's bottleneck analysis has particular resonance in the claims context. Even where a substantial proportion of model outputs are ultimately discounted through expert review, the remaining volume of genuine vulnerabilities can still put considerable pressure on remediation teams, engineering resources, and change management processes.  Claims operations face a structural version of this same challenge: AI triage tools may correctly identify a large volume of claims for automated settlement, but the residual claims that require human assessment, whether because the AI flags uncertainty, or because a consumer challenges an automated decision, must be handled by a team with sufficient capacity and expertise to do so meaningfully. Firms that have reduced their human claims-handling capacity in parallel with deploying AI triage should review whether the remaining human resource can in practice fulfil the oversight and challenge function that the regulatory framework requires.