A Reference Architecture for AI in Large Organisations

A reference architecture is a shared answer to the question of what an organisation's AI estate should consist of, so that each new use case extends a known structure rather than inventing one. This chapter presents a layered reference architecture for AI in large enterprises and public authorities, its four central patterns, its mapping to the AI Act roles, and a declarative model of it that frames every later chapter.

Figure 4.1: The reference architecture stacks five functional layers from the data foundation to applications and channels, with a cross-cutting control plane for identity, policy, observability, and evidence.

Why a reference architecture

A reference architecture earns its cost when the alternative is twenty teams making twenty incompatible decisions. Kreuzberger, Kühl, and Hirschl (2023) derived an MLOps architecture from literature and practitioner interviews, and found a stable set of components recurring across organisations: a source code repository, CI/CD, workflow orchestration, a feature store, training infrastructure, a model registry, metadata stores, model serving, and monitoring (68, Kreuzberger et al., 2023). Huyen (2022) describes the same components as an iterative design process (74, Huyen, 2022). Both accounts, however, predate the shift that Zaharia et al. (2024) describe, in which the unit of delivery becomes a compound AI system that combines models, retrieval, tools, and control logic (63, Zaharia et al., 2024). The reference architecture in this chapter therefore takes the MLOps components as its foundation and adds the layers that compound systems and agents require.

Architecture as Code argued that a reference architecture is useful only when it is codified, versioned and checked against the running estate (72, Nordqvist, 2025). Brown's C4 model describes software at four levels, system context, containers, components, and code (39, Brown, n.d.); the layers below correspond to the container level of a C4 model of the AI platform.

The six layers

Value flows upward from data to applications, and control flows inward from the control plane to every layer.

Data foundation

The data foundation provides the data that models are trained on and grounded in, in a form that is owned, documented, and traceable: data products with contracts and named owners, a lineage catalogue, a feature store, and vector indexes for retrieval. It is where the GDPR obligations of lawful basis and purpose limitation are enforced (2, European Union, 2016), and where the data governance requirements of Article 10 of Regulation (EU) 2024/1689, the AI Act, are satisfied for high-risk systems (1, European Union, 2024). Chapter 6 develops it.

Model and feature layer

The model and feature layer turns data into approved models through experiment tracking, training pipelines, fine-tuning infrastructure, the model registry, and the evaluation service. The registry is the only place from which a model may be promoted to serving, and every entry carries a model card, evaluation results, and lineage back to the data foundation (Chapter 7).

Serving and inference layer

The serving and inference layer makes models callable, through internal model serving, connectors to external model providers, and the AI gateway through which every call passes. Separating this layer from the orchestration layer above it is the single most important structural decision in the architecture, because it allows the organisation to change which model answers a request, or where that model runs, without changing any application. It is also where the sovereignty decisions of Chapter 13 are implemented: a routing rule in the gateway decides whether a request may leave the European Economic Area.

Orchestration and agent layer

The orchestration and agent layer composes models, retrieval, and tools into systems that accomplish tasks. It contains the workflow engine, the agent runtime, the tool registry, the prompt repository, and the guardrails. Bounded autonomy is enforced here: an agent's permission manifest is evaluated before any tool call is executed, by a policy engine that the agent does not control. Chapters 10 and 11 describe retrieval-augmented generation and agents.

Agents and connectors

Agents add a second interface to the estate, because every connector is both a data path, through which content reaches a model, and an action path, through which a model's output reaches a system of record. The Model Context Protocol has become the common way to expose such connectors, as servers that publish tools, resources and prompts to any compatible client (36, Anthropic, 2024), and its own security guidance addresses the authorisation, consent and token-handling mitigations to those who implement MCP and operate its servers (229, MCP, 2025).

Four components make this layer governable.: a tool registry of permitted connectors with owners, data classes and actions; a skills catalogue of reviewed instructions (Chapter 8); an agent identity registry giving each agent its own workload identity; and an approval service with a stop path the agent cannot reach. Three rules follow: no connector is used unless it is registered; the platform, not the agent, holds the credentials; and every tool call passes the same policy decision point as a model call, so that the risks the OWASP agentic list describes, from excessive agency to tool misuse and identity abuse, meet a control outside the model (102, OWASP, 2025). Where suite vendors offer agent registries of their own (149, Lamanna, 2025), the organisation's registry should import their inventory rather than run a parallel list.

Application and channel layer

The application and channel layer is what people and other systems interact with: assistants, decision-support applications embedded in case management, and APIs exposed to partners. Applications never call models directly, so that oversight, logging, and policy are applied uniformly. Human oversight becomes visible here (Chapter 12).

Cross-cutting control plane

The control plane surrounds the stack. Identity gives every workload, model, pipeline, and agent an identity of its own and propagates the identity of the human on whose behalf a request is made. Policy evaluates every significant action against rules held in a repository, using an engine such as Open Policy Agent, whose Rego rules pipelines, gateways, and orchestrators can all query (35, Open Policy Agent, n.d.). Observability collects traces, metrics, and logs; the OpenTelemetry semantic conventions for generative AI define standard attributes for model requests, so that AI calls appear in the same traces as the rest of the estate (34, OpenTelemetry, n.d.). Evidence stores the evaluation results, approvals, logs, and generated documentation that AI Governance, Chapter 11 assembles into evidence packs. The control plane is thus the architectural expression of policy as code and evidence as code (72, Nordqvist, 2025).

Four patterns that carry the architecture

Within the layers, four patterns recur in every implementation I have seen succeed. Figure 4.2 shows how they interact around the gateway.

Figure 4.2: Every model call passes through the AI gateway, which authenticates the caller, consults the policy decision point, resolves the approved model and prompt version, routes to internal or external serving, and emits telemetry that becomes evidence.

The AI gateway

The AI gateway is a single, mandatory entry point for model calls. It authenticates the calling workload and the human behind it, asks the policy decision point whether the request is permitted, resolves the approved model and prompt versions, applies input and output guardrails, routes the request, enforces budgets and emits a trace. With it, the organisation has a single place to enforce a prohibited use, to switch providers and to produce the logs that Article 26(6) of the AI Act requires deployers of high-risk systems to retain for at least six months (1, European Union, 2024).

The model registry

The model registry is the authoritative list of models the organisation may use, including external foundation models registered with their provider, version, and documentation. Each entry moves through stages, typically candidate, evaluated, approved, and retired, and pipelines with the evaluation gates of Chapter 7 perform every promotion, never a person by hand. The registry is the system of record for the AI inventory that ISO/IEC 42001 expects and that AI Governance, Chapter 9 codifies.

The evaluation service

The evaluation service runs evaluation suites against candidate models, prompts, and compound systems, before promotion and continuously in production. It owns the golden datasets, the acceptance thresholds, and the champion and challenger comparison, and it writes its results both to the registry, where they gate promotion, and to the evidence store, where they become the record that Articles 9 and 72 of the AI Act require for high-risk systems (1, European Union, 2024). Chapter 20 develops it.

Prompt and policy repositories

Prompts and policies are configuration, and configuration belongs in version control. The prompt repository holds every system prompt and template with a version, an owner, and the evaluation results that accompanied its approval; the policy repository holds the rules for classification, data use, model approval, agent permissions, and routing. Both are consumed at runtime by reference, so that a change is a reviewed commit followed by a pipeline run, not an edit in a console. For AI the case for configuration as architecture (72, Nordqvist, 2025) is sharper, because a prompt change can alter behaviour as much as a model change.

Mapping the architecture to AI Act roles

Article 3 defines the provider, who develops a system and offers it under its own name (Article 3(3)), and the deployer, who uses a system under its own authority (Article 3(4)). In the reference architecture, a provider owns the data foundation, the model and feature layer, and the serving layer, and carries the obligations of Articles 9 to 15 for high-risk systems; a deployer owns the orchestration and application layers, and carries the obligations of Article 26 (1, European Union, 2024).

Most large organisations are both, and under Article 25 rebranding a high-risk system, changing it significantly, or redirecting it to a high-risk purpose turns a deployer into its provider (1, European Union, 2024). The registry entry for every model therefore records which role the organisation occupies for it, and the control plane applies the corresponding obligations (Chapter 3).

Public authorities carry one more mapping. Under Article 49(3), a public-authority deployer must register its use of an Annex III high-risk system in the EU database before putting it into service, and under Article 27 must first complete a fundamental rights impact assessment (1, European Union, 2024). Both are produced from the registry and evidence store, which requires a complete inventory.

Expressing the architecture as a declarative model

The reference architecture is kept as a declarative model, as Architecture as Code keeps architecture models in Structurizr DSL or equivalent formats (72, Nordqvist, 2025). Pipelines consume it to check that a deployed system uses only components the model declares, and documentation is generated from it.

# architecture/reference-model.yaml - declarative model of the AI reference architecture
apiVersion: ai-architecture.example.org/v1
kind: ReferenceArchitecture
metadata: {name: enterprise-ai-platform, version: 2.0.0, owner: ai-platform-team}
layers:
  - id: data-foundation
    components: [data-products, lineage-catalogue, feature-store, vector-index]
  - id: model-and-feature
    components: [experiment-tracking, training-pipelines, model-registry, evaluation-service]
  - id: serving-and-inference
    components: [ai-gateway, internal-model-serving, external-model-connectors]
  - id: orchestration-and-agents
    components: [workflow-engine, agent-runtime, tool-registry, prompt-repository, guardrails]
  - id: application-and-channel
    components: [assistants, decision-support-apps, partner-apis]
controlPlane:
  identity: workload-identity-with-oidc
  policy: {engine: open-policy-agent, repository: ai-platform/policies}
  observability: {standard: opentelemetry-genai-semantic-conventions}
  evidence: {store: evidence-bucket-eu-north, retentionMonths: 6}
components:
  ai-gateway:
    responsibilities: [authenticate, authorise, route, apply-guardrails, log]
    dependsOn: [model-registry, prompt-repository, policy-engine]
  model-registry:
    stages: [candidate, evaluated, approved, retired]
    promotionBy: pipeline-only
aiActRoles:
  provider: {ownsLayers: [data-foundation, model-and-feature, serving-and-inference]}
  deployer: {ownsLayers: [orchestration-and-agents, application-and-channel]}

A team adapting the model records deviations in an AI Decision Record, but should not remove the gateway, the registry, the evaluation service, or the control plane, because each later chapter assumes they exist.

Worked example: a component manifest for one use case

Illustrative example. Each use case declares which components of the reference model it uses, in which role, and with what configuration, and a pipeline check fails if the use case bypasses the gateway or names a component the platform does not offer. Hansa Mutual's manifest for its claims assistant (AIDR-0021) pins enterprise-ai-platform@2.0.0 and binds the assistant to a gateway route, a versioned system prompt and guardrail set, a general-purpose model in the registry, a vector index whose entitlements are carried from source, and an evaluation suite. Three choices deserve comment. Its aiActRole block records that Hansa deploys the general-purpose model but provides the compound system it has assembled, the split Chapter 5 explains. Its markets block records that one assistant serves six insurance markets and is processed in Germany and at the Polish service centre, so per-market overlays, not copies of the manifest, carry what differs by country. And its deviation from the reference model's six-month retention to 120 months, for DORA and claims disputes, is declared and linked to AIDR-0022, so that it is a recorded adaptation rather than drift.

Deployment variants

The layers say what an AI estate consists of, not where it runs. Four deployment variants recur, each moving the boundary between what the organisation operates and what it buys; Chapters 13 and 14 treat placement in detail.

Layer	Cloud	On-premises	Hybrid	Sovereign
Data foundation	Cloud data platform in a pinned region	Own data centre	Where the data's classification requires	European operator under European law
Model and feature	Managed training and registry services	Own pipelines and registry	One registry, deployable everywhere	Own pipelines on rented dedicated capacity
Serving and inference	Provider APIs and managed endpoints	Own accelerators and open-weight models	Self-hosted and cloud tiers behind one gateway	Dedicated capacity, own weights
Orchestration and agents	Managed agent platform or own runtime	Own runtime	Own runtime, routing by data class	Own runtime
Control plane	Organisation's, with provider logs exported to it	Organisation's	Organisation's, spanning both	Organisation's, with jurisdiction attributes verified

Table 4.1: What each deployment variant changes in the reference architecture.

In the cloud variant model versions retire on the provider's schedule, so the registry carries deprecation dates and the evaluation service re-runs when a version changes. On premises, the organisation owns every patch, capacity shortfall and serving vulnerability. The hybrid variant, which most large organisations reach, is where the gateway and single registry pay for themselves; agentic work tools are its hardest case, and Chapter 15 shows how routing by classification is enforced outside the tool. The sovereign variant resembles on-premises, except that the control plane records the jurisdiction, operator and key custody of each location as attributes policy can test, using a vocabulary such as the Gaia-X framework where available (22, Gaia-X, n.d.).

In every variant the application layer stays the same, because applications call logical model names through the gateway, and the control plane remains the organisation's own, because the obligations it evidences remain the organisation's own. Moving the evidence store or policy repository into a provider's console moves accountability with it.

Adapting the reference architecture

Three adaptations cover most organisations: a group runs one control plane and gateway per legal entity with a shared registry; a financial institution adds a model risk function as a mandatory approver in the promotion workflow; and a public authority keeps the data foundation within national or EEA infrastructure and generates its public algorithm register from the registry. In each case the layers stay the same; only the boundaries of ownership and jurisdiction move.

Multi-jurisdiction overlays

A group operating in several countries meets a harder version of the first adaptation, because its entities sit under different laws. Hansa Mutual's EU entities are subject to the AI Act and DORA (3, European Union, 2022); its Norwegian entity follows the AI Act only once it is incorporated into the EEA Agreement (241, EFTA, n.d.); and its UK and Swiss entities sit outside both, under the United Kingdom's sector-led approach (242, DSIT, 2023) and Swiss law respectively. Copying the platform per country would multiply cost and drift, so the architecture layers three files: the shared reference model, a group baseline, and one overlay per market recording regime (which instruments apply), placement (where logs and evidence must be kept) and timing (when an obligation starts to apply). One rule keeps the structure sound: an overlay may add or tighten controls but never relax the group baseline, and a pipeline check fails any overlay that tries.

Worked example: a market overlay for a non-EU entity

Illustrative example. Hansa Mutual's UK subsidiary uses the group platform in Germany. Its overlay states that the AI Act applies only as group policy, names the UK instruments that apply instead, keeps the entity's logs and evidence in the United Kingdom, and lists the components no overlay may remove.

# architecture/overlays/markets/uk.yaml - Hansa Mutual (illustrative)
apiVersion: ai-architecture.example.org/v1
kind: MarketOverlay
metadata:
  base: enterprise-ai-platform@2.0.0
  market: UK
  entity: hansa-mutual-uk
  owner: uk-entity-chief-risk-officer
regime:
  aiAct: group-policy
  dora: not-applicable
  dataProtection: [uk-gdpr, data-protection-act-2018]
  supervisors: [PRA, FCA]
placement:
  gatewayLogs: {region: uk-south}
  evidence: {region: uk-south, retentionMonths: 120}
  processingAllowedIn: [UK, DE]
mayNotRelax: [ai-gateway, model-registry, evaluation-service, control-plane]
reviewWhen:
  - UK legislation on AI changes the entity's obligations
  - the adequacy decision for the United Kingdom is amended or expires

The entity relies on the group's AI Act controls as policy, while UK data protection law, in which the Data Protection Act 2018 supplements the UK GDPR (248, UK Parliament, 2018), and the UK supervisors' expectations are the obligations it must actually evidence. The Norwegian overlay differs in one field, the AI Act's status, which becomes a legal obligation when incorporation takes effect.

A reduced variant for a smaller authority

An authority that mostly procures needs a reduced variant that keeps the controls and drops the machinery of building; the test for each component is whether the organisation's own obligations depend on it. The gateway stays, bought or run from an open-source project, because it is the only place where logging, redaction, routing and budgets apply to every AI call. The registry stays, even as a version-controlled file per system, because the inventory, the public algorithm register and the Article 49 registration are generated from it. Evaluation stays as acceptance suites the authority runs against each supplier release, and the control plane is built from the identity provider, logging platform and archive the authority already operates. Experiment tracking, training pipelines and feature stores, by contrast, belong to the provider of a procured system, and the contract carries what they would have produced: model documentation, evaluation results, release notes and the logs the deployer must retain. Nordvik Region writes its version as an overlay on the pinned base model.

# architecture/overlays/nordvik-region.yaml - Nordvik Region (illustrative)
apiVersion: ai-architecture.example.org/v1
kind: ReferenceArchitectureOverlay
metadata:
  base: enterprise-ai-platform@2.0.0
  organisation: nordvik-region
  decisionRecord: AIDR-0002
placement:
  data-foundation: {jurisdiction: national, operator: nordvik-datacentre}
  external-model-connectors: {allowedJurisdictions: [eea], personalDataAllowed: contracted-eea-providers-only}
addComponents:
  - {id: public-algorithm-register, layer: application-and-channel, generatedFrom: [model-registry, decisions]}
  - {id: fria-workflow, layer: control-plane, requiredFor: high-risk}
controlPlaneOverrides:
  evidence: {retentionMonths: 120, archiveTo: regional-archive-service}
approvals:
  modelPromotion: {highRisk: [head-of-case-management, data-protection-officer]}

Practical guide: deriving your own variant from the reference architecture

Purpose. Produce a checkable variant that fits the organisation, rather than an aspiration copied from a larger one.

Roles. Chief architect (owner), platform lead, security architect, data protection officer, and a representative of procurement.

1. List every AI system in the inventory with its usage mode and whether the organisation builds or procures it.
2. For each layer, mark whether the organisation operates it, buys it as a service, or relies on a supplier's product, using Table 4.1.
3. Keep the gateway, the registry, evaluation and the control plane in every case; drop or buy any other component on which no obligation depends.
4. Write the result as an overlay on the pinned reference model, never as a copy, with the reasons in an AI Decision Record; add one market overlay per entity in another jurisdiction.
5. Add a pipeline check that compares each system's component manifest with the variant and fails on a bypassed gateway or an undeclared component.
6. Review the variant when a usage mode, deployment variant or jurisdiction is added, and at least once a year.

Definition of done. The variant and its overlays are in version control with a decision record; every production AI system has a component manifest that passes the pipeline check; and the inventory, the evidence store and, for a public authority, the public register are generated from the same registry.

How the later chapters build on the reference architecture

Table 4.2 shows where each part is developed; AIG marks chapters of AI Governance.

Layer or pattern	Chapters that develop it
Data foundation	6; AIG 5
Model and feature layer, registry	5, 7, 8 and 9
Serving and inference	10, 13 to 15, 22
Orchestration, agents and connectors	11, 12, 18 and 19
Gateway and evaluation service	12, 14, 20 and 21
Prompt and policy repositories	10; AIG 8 and 9
Control plane	16 and 17; AIG 11

Table 4.2: Where each part of the reference architecture is developed.

Across the usage modes

The table shows which layers the organisation operates in each usage mode of Chapter 2 (usage-mode matrix), and how much of the control plane reaches it.

Mode	Layers the organisation operates	Control-plane coverage
Mode 0 – Ambient AI	None; the feature lives in supplier software	Contract terms and an inventory entry
Mode 1 – Conversational assistants	None beyond the channel	Single sign-on, data loss prevention, the gateway if used through an API
Mode 2 – Embedded workplace assistants	None of its own	Tenant identity, permissions and the suite's audit logs
Mode 3 – Agentic work tools	Orchestration, as an agent runtime on a workstation or sandbox	Identity, tool permissions; the gateway where the tool accepts a custom endpoint
Mode 4 – Built AI applications	All five layers, partly through external model connectors	The full control plane
Mode 5 – Autonomous production agents	All five, with the agent runtime and tool registry at the centre	The full control plane, with a policy decision on every tool call

The gateway is the hardest gap in Modes 1 to 3, so where a tool lets traffic be routed through it, route it. The registry, by contrast, covers every mode, and the agent identity registry should list Mode 2 and Mode 3 agents beside the Mode 5 agents the platform runs.

Nordvik Region and Hansa Mutual

Nordvik Region runs the reduced variant through the overlay recorded in AIDR-0002: because it mostly procures, its leverage lies in the upper layers, the gateway and the evidence it can demand by contract. Its deployment is mixed: the triage system in its regional data centre, the citizen-service assistant on capacity a European hosting provider reserves for it, and the staff assistant through a European cloud provider with customer-held keys.

Hansa Mutual, by contrast, runs the full reference architecture as a shared group platform and is provider of the compound systems it assembles. Its deployment is hybrid, with self-hosted and fallback models in Frankfurt and hosted models behind the same gateway, and its orchestration layer carries the full agent and connector components because its claims-triage and IT-operations agents act on production systems. Where Nordvik's boundary is national, Hansa's are many.

Summary

The reference architecture has five functional layers, from the data foundation to application and channel, surrounded by a control plane for identity, policy, observability, and evidence. Four patterns carry it: the AI gateway as the mandatory entry point for model calls, the model registry as the system of record, the evaluation service as the source of promotion decisions and evidence, and the prompt and policy repositories. Agents add a connector layer in which every tool is registered, every credential is held by the platform and every tool call meets the same policy decision as a model call.

The layers stay the same across deployment variants while the boundary between what the organisation operates and what it buys moves, and adaptations are written as overlays on a pinned reference model. Because Article 25 can move an organisation from deployer to provider, the registry records the role for every model. Chapter 5 extends Architecture Decision Records to AI and shows how the architecture repository becomes the single source of truth for the technical documentation that Article 11 and Annex IV of the AI Act demand.

Sources

1. Kreuzberger, D., Kühl, N. & Hirschl, S. (2023). Machine Learning Operations (MLOps): Overview, Definition, and Architecture. IEEE Access, 11, pp. 31866–31879. (68, Kreuzberger et al., 2023)
2. Huyen, C. (2022). Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications. Sebastopol, CA: O'Reilly Media. ISBN 978-1-0981-0796-3. (74, Huyen, 2022)
3. Zaharia, M., Khattab, O., Chen, L., Davis, J. Q., Miller, H., Potts, C., Zou, J., Carbin, M., Frankle, J., Rao, N. & Ghodsi, A. (2024). The Shift from Models to Compound AI Systems. Berkeley Artificial Intelligence Research blog, 18 February 2024. (63, Zaharia et al., 2024)
4. Nordqvist, G. (2025). Architecture as Code. The Transformed Architecture, Book 1, published at aac.geon.se. (72, Nordqvist, 2025)
5. Brown, S. The C4 model for visualising software architecture.. (39, Brown, n.d.)
6. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
7. Anthropic. Model Context Protocol Specification. November 2024, revised 2025. (36, Anthropic, 2024)
8. Model Context Protocol (2025). Security Best Practices. MCP specification version 2025-11-25. (229, MCP, 2025)
9. OWASP Foundation (2025). OWASP Top 10 for Agentic Applications for 2026. OWASP GenAI Security Project, 9 December 2025. (102, OWASP, 2025)
10. Lamanna, C. (2025). Microsoft Agent 365: The control plane for AI agents. Microsoft 365 Blog, 18 November 2025. (149, Lamanna, 2025)
11. Open Policy Agent Contributors. Open Policy Agent Documentation. Cloud Native Computing Foundation. (35, Open Policy Agent, n.d.)
12. OpenTelemetry Authors. OpenTelemetry Documentation, including the Semantic Conventions for Generative AI systems. Cloud Native Computing Foundation. (34, OpenTelemetry, n.d.)
13. Gaia-X European Association for Data and Cloud AISBL. Gaia-X: A Federated Secure Data Infrastructure.. (22, Gaia-X, n.d.)
14. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). Official Journal of the European Union, 27 December 2022. (3, European Union, 2022)
15. European Free Trade Association. EEA-Lex factsheet 32024R1689: the Artificial Intelligence Act. EFTA. (241, EFTA, n.d.)
16. Department for Science, Innovation and Technology and Office for Artificial Intelligence (2023). A pro-innovation approach to AI regulation (CP 815). UK Government, 29 March 2023. (242, DSIT, 2023)
17. United Kingdom (2018). Data Protection Act 2018 (c. 12), with the UK General Data Protection Regulation. (248, UK Parliament, 2018)
18. European Parliament and Council of the European Union. Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Official Journal of the European Union, 4 May 2016. (2, European Union, 2016)