From Pilot to Production: Scaling AI Across the Organisation
Most organisations that I meet have no shortage of AI pilots; they have a shortage of AI in production. A demonstration that impressed a steering committee in March is still a demonstration in November, and the team that built it has moved on. This chapter explains why pilots stall and how to codify the path from idea to governed production service, treating the platform, the intake process, and the readiness review as versioned artefacts rather than meetings, so that the second use case is cheaper than the first and the tenth is routine.
Figure 23.1: The industrialisation path. Pilots that run on the shared platform pass through a codified readiness gate into production; pilots built beside the platform fall into the pilot trap.
The pilot trap
The pilot trap is the state in which an organisation can start AI initiatives but not finish them. Its symptoms are proofs of concept with no owner, model code in a notebook on a laptop, and a sponsor who no longer answers emails about funding. Paleyes, Urma and Lawrence (2022) surveyed published case studies of machine learning deployment and found that practitioners face difficulties at every stage of the deployment workflow (67, Paleyes et al., 2022). The trap is not a failure of talent but of architecture.
Why pilots stall
A pilot optimises for showing that a capability is possible, and defers everything production demands. Sculley and colleagues (2015) catalogued the technical debt this creates: entanglement between features, undeclared consumers of model output, data dependencies that cost more than code dependencies, glue code around a small core of learning logic, and configuration that can far exceed the traditional code in volume (43, Sculley et al., 2015). A pilot accumulates all of these in weeks, and the debt falls due at the move to production. Each of the differences between machine learning and conventional software engineering that Amershi and colleagues observed at Microsoft, described in Chapter 7, becomes a stalling point: the pilot used a data extract that cannot be refreshed lawfully, fine-tuned a model the organisation has no licence to deploy, or bundled retrieval, prompting, and post-processing into one untestable script (65, Amershi et al., 2019).
With generative AI, a business unit wires a hosted model to a document store and demonstrates a policy chatbot that works because the demonstrator chose the questions. Nobody has defined an evaluation set or asked whether the organisation is a deployer under Regulation (EU) 2024/1689, the AI Act (1, European Union, 2024). The pilot stalls not because it is wrong but because nobody can say how right it is.
Pilots built beside the estate
The deeper cause is that most pilots are built beside the enterprise estate, in a sandbox with its own identity model and logging and no connection to the change process. Retrofitting controls costs more than building with them. Architecture as Code (Chapter 5) argued that the pipeline, not the environment, is the unit of trust: a system is production-ready when the path that builds, tests, and deploys it is itself versioned and reviewed (72, Nordqvist, 2025). The question to ask of a pilot is therefore whether it was built on a path that can carry it to production without being rebuilt.
Across the usage modes
The pilot trap and the path in Figure 23.1 apply mainly to Mode 4 and Mode 5, where the organisation builds and operates the system. In Mode 0 there is no pilot, because the supplier decides when the feature arrives. In Modes 1 to 3 the organisation procures a finished tool, and moving from a trial group to the whole workforce is a rollout with its own readiness steps.
| Usage mode | What "production" means | Readiness steps before scaling |
|---|---|---|
| Mode 0 Ambient AI | A supplier enables a feature for everyone | Detect the change, keep or disable it, update guidance |
| Mode 1 Conversational assistants | Licensed access for all intended staff | Acceptable-use policy, AI literacy measures under Article 4, tenant settings |
| Mode 2 Embedded workplace assistants | Add-on enabled per user group | Permission and oversharing clean-up, sensitivity labels, staged enablement |
| Mode 3 Agentic work tools | Tool approved for defined teams | Command permissions, sandboxing, review rules for agent-made changes |
| Mode 4 Built AI applications | A governed service on the platform | Golden path, intake record, evaluation set, readiness review, go/no-go |
| Mode 5 Autonomous production agents | An agent acting with delegated authority | As Mode 4, plus authority limits, kill switch, staged autonomy |
A Mode 2 rollout stalls when the assistant surfaces documents whose permissions were always too broad, so its readiness work sits in the estate. The rest of this chapter is written for Modes 4 and 5.
Platform thinking and golden paths for AI
The alternative to the pilot trap is a platform. Skelton and Pais (2019) describe a platform team as one that provides internal services which reduce the cognitive load of stream-aligned teams, offered as a product rather than imposed as a mandate (78, Skelton & Pais, 2019). For AI the platform is the set of capabilities every use case would otherwise build for itself: model access, evaluation, guardrails, observability, and evidence collection. Chapter 4 introduced these as the control plane of the reference architecture; here they are the reason the second use case is cheaper than the first.
The AI platform as a product
A platform is a product when teams choose it as the easiest route to production, and the golden path makes it so: a documented, templated, supported way of building a particular kind of AI service with the controls already wired in. A team that follows the path for a retrieval-augmented generation (RAG) assistant inherits the gateway configuration, evaluation harness, logging schema, and deployment pipeline; a team that leaves it must prove equivalence at the readiness review. The path lives in version control as a template repository, and platform teams measure success by how often it is used and how little the review has to reject.
Worked example: golden path template manifest
Illustrative example. The manifest below sits at the root of Hansa Mutual's golden path template for a RAG assistant. It declares what the template creates, which platform services it binds to, and which settings a team may change without leaving the path.
# golden-paths/rag-assistant/template.yaml
golden_path:
name: rag-assistant
version: 4.2.0
owner: ai-platform
intended_for: internal assistants that answer from governed document collections
not_for: [fully automated decisions, customer-facing advice without review]
creates:
- use-cases/intake/{use_case_id}.yaml
- decisions/{use_case_id}/aidr.md
- evaluation/{use_case_id}/golden-set.jsonl
- guardrails/{use_case_id}/policy.yaml
- readiness/{use_case_id}/checklist.yaml
- .pipeline/deploy.yaml
binds_to:
gateway: {route_prefix: "/{use_case_id}", default_model: medium-eu-hosted-32b}
evaluation_service: {suite: rag-baseline-v3, block_on_failure: true}
logging: {schema: inference-log-v2, retention_days: 180}
team_may_change:
- prompts and few-shot examples
- retrieval top_k between 2 and 8
- additional guardrail rules (stricter only)
leaving_the_path:
- model access outside the gateway
- logging retention below 180 days
- disabling evaluation gate
- consequence: readiness review requires evidence of equivalence for each item
The lists team_may_change and leaving_the_path are the core of the manifest: they tell a team, before it writes any code, which decisions are its own and which carry an evidence burden. The not_for line tells a team planning automated decisions at once that it needs a different path.
Reusable components: gateway, evaluation, guardrails
Three components return the most value when shared: the AI gateway (Chapter 12), the single point of model access and the one place to swap a model; the evaluation service (Chapter 20), which records test results as evidence; and the guardrail layer (Chapter 16). Building these complicated subsystems once turns a collection of pilots into a programme.
Use-case intake and prioritisation
Without a front door, use cases arrive through whichever executive is loudest. Intake is where the organisation decides what to build, in which order, and under which obligations, and it is the first point at which an AI Decision Record (Chapter 5) can be opened.
The intake record as code
The intake record is a short structured file in the use-case portfolio repository that makes the decision to proceed reviewable and seeds the risk classification. AI Governance, Chapter 8 treats that classification as code: the intake record is the descriptor over which a policy evaluates the AI Act's categories, and the resulting risk class decides which assessments and readiness items the gate will demand.
# use-cases/intake/claims-triage-assistant.yaml
intake:
id: UC-2026-014
title: Claims triage assistant
sponsor: Head of Claims Operations
value_hypothesis:
metric: median_triage_time_hours
baseline: 48
target: 8
decision_impact: recommends priority; handler decides
data_sources:
- name: claims_intake_system
personal_data: true
lawful_basis: contract
ai_act:
likely_role: provider_and_deployer # built on a vendor model
annex_iii_candidate: false
reasoning: Recommendation only; handler remains the decision maker.
platform:
golden_path: rag-assistant
prioritisation: {value: 4, feasibility: 3, risk: 2, reuse_potential: 4}
Every field does work later: the value hypothesis becomes the benefit that AI Governance, Chapter 16 measures after go-live. The data sources seed the data protection impact assessment (DPIA) required under Article 35 of the GDPR (2, European Union, 2016), which Article 26(9) of the AI Act expects deployers of high-risk systems to carry out using the provider's instructions for use. The role assessment determines which obligations the readiness review checks.
Prioritisation criteria
Four criteria have proved sufficient: business value, technical feasibility, risk, and reuse potential. The fourth is the one most organisations omit and the one that most determines whether a programme scales, because a use case that forces the platform to add a capability the next five also need is worth more than its own business case suggests. Conversely, a use case needing a bespoke model outside the gateway, with no evaluation set, should score poorly on feasibility. Scores are committed rather than presented, which prevents quiet re-ranking between meetings.
Production readiness reviews
The production readiness review is the gate between a pilot and a service. In the pilot trap it is either absent or a slide-deck meeting that checks what the presenter chose to present. The alternative is a codified checklist whose items are verified by evidence in the repository, most of them automatically.
The review as a gate, not a meeting
The ML Test Score, which Chapter 7 applies as the model's readiness gate, offers a scored rubric that makes production readiness measurable and repeatable (66, Breck et al., 2017): a review that produces a score and a list of failing checks can be re-run after each fix, whereas one that produces minutes cannot. The automated part verifies the presence and freshness of required artefacts such as the AI Decision Record, evaluation results, and runbook. The human part, involving the architect, the data protection officer, security, and the business owner, considers whether the intended purpose is honestly described, whether the oversight design is credible, and whether the organisation is willing to be accountable for the output.
A readiness checklist aligned to AI Act obligations
Where the use case is, or may be, a high-risk AI system, the checklist maps to the deployer obligations in Article 26 of the AI Act.
| Article 26 obligation | Architectural control | Evidence in repository |
|---|---|---|
| 26(1) Use in accordance with the instructions for use | Intended purpose and limits enforced in configuration | AI Decision Record, gateway configuration |
| 26(2) Human oversight by competent, trained persons with authority | Oversight role model, interruption path | RBAC definition, training records |
| 26(4) Input data relevant and representative, where the deployer controls it | Data contract and input validation | Validation results |
| 26(5) Monitor operation and inform provider and authority of risks | Drift and quality monitoring routed to incidents | Alert rules, incident runbook |
| 26(6) Retain logs for at least six months | Retention policy on inference logs | Retention configuration |
| 26(7) Inform workers' representatives before workplace use | Consultation recorded as a gate | Consultation record |
| 26(8) Public authority deployers register in the EU database | Registration step in the release pipeline | Registration reference |
| 26(11) Inform persons subject to the system's decisions | Transparency notice in the channel | Notice text under version control |
Beside Article 26, Article 4 requires AI literacy measures for staff, verified through training records (AI Governance, Chapter 14). Article 27 requires deployers that are bodies governed by public law, or private operators providing public services, to carry out a fundamental rights impact assessment (FRIA) before first use of a high-risk system, which makes it a gate item for public-sector readers. Article 50 imposes transparency duties on providers of systems that interact directly with natural persons or generate synthetic content, and on deployers of certain others, such as systems that generate deep fakes (1, European Union, 2024).
The checklist is a file, abridged here to three checks:
# readiness/checklist.yaml
production_readiness:
use_case: UC-2026-014
gate_version: 3
checks:
- id: ADR-PRESENT
description: AI Decision Record approved
evidence: decisions/UC-2026-014/adr.md
automated: true
- id: LOG-RETENTION
description: Inference logs retained for at least 180 days
evidence: platform/logging/retention.yaml
automated: true
regulatory_reference: AI Act Article 26(6)
- id: WORKER-CONSULTATION
description: Workers' representatives informed before go-live
evidence: decisions/UC-2026-014/consultation.md
automated: false
regulatory_reference: AI Act Article 26(7)
outcome: pending
A pipeline stage blocks deployment until each item's evidence exists and is newer than the last model or prompt change, or an exception is recorded with an owner and an expiry. AI Governance, Chapter 9 describes such exceptions as records under version control, each with a compensating control, so that an expired exception fails the gate like a missing artefact.
When Hansa Mutual ran the gate for UC-2026-014, which the intake record classified as not high-risk, it applied the Article 26 rows as good practice and marked the public-authority obligations as not applicable. The run recorded seventeen passes, two passes with exception, and one failure. EXC-112 covered five untrained handlers, compensated by an access control that blocks them; EXC-113 covered a fallback to manual triage tested only in staging, with a production drill scheduled. The single failure belonged to the business owner, which is common: the controls were in place, but nobody had agreed the baseline that defines success.
Worked example: go/no-go decision record
Illustrative example. The readiness report informs the decision but does not make it. Hansa Mutual records the go-live decision as an AI Decision Record in the style of Chapter 5, so that the decision log shows who accepted the residual risk and on what conditions.
---
adr_id: AIDR-0034
title: Go-live decision for UC-2026-014 Claims triage assistant
status: Accepted
date: "2026-10-21"
relates_to: [AIDR-0063, readiness/UC-2026-014/review-2026-10-14.md]
deciders: [Head of Claims Operations, Head of AI Platform]
next_review_due: "2027-01-21"
---
# AIDR-0034: Go-live decision for UC-2026-014 Claims triage assistant
Context: readiness review 2026-10-14; the failed business-owner check passed
on re-run 2026-10-20 after finance agreed the 48-hour baseline.
Options: 1. Full go on 2026-11-01. 2. Conditional go: property claims only,
30 % of handlers, six weeks. 3. No go until EXC-112 and EXC-113 close.
Decision: Option 2, starting 2026-11-04.
Conditions: EXC-112 and EXC-113 closed before expansion; gateway budget cap
EUR 4,000 per month during the controlled rollout.
Stop criteria (any one triggers rollback to manual triage):
- Grounding on weekly sampled cases below 0.90 for two consecutive weeks.
- Handler override rate above 35 % in any week.
- Any confirmed personal data from another claim in output.
Accountability: Head of Claims Operations accepts the residual risk.
The record keeps the rejected options, and its stop criteria are measurable from the observability configuration verified at the gate, so a rollback depends on a threshold rather than a meeting. Accountability sits with the business owner, as in the operating model of AI Governance, Chapter 12.
Closing the pilots that do not proceed
The gate has a second outcome that organisations rarely design for: a pilot that will not proceed. A rejected pilot often holds production data extracts, live credentials, and an index whose permissions were copied once and never updated: an attack surface of the kind Chapter 16 describes, and a copy of personal data that the storage-limitation principle in Article 5(1)(e) of the GDPR does not allow the organisation to keep without purpose (2, European Union, 2016). Close a pilot as deliberately as you launch a service: revoke credentials, delete extracts and indexes with a recorded confirmation, and archive the decision record that ended it. The golden path makes this cheap, because its template lists the resources it created.
Scaling across business units and agencies
Industrialising fifty use cases across a group or a national administration is an operating model problem, not a project.
Federation without fragmentation
The pattern that works is federated ownership on a shared platform: the platform team owns the shared components, golden path templates, and readiness checklist; business units own their use cases, data, evaluation sets, and accountability as deployers. The golden path is the contract between them. Forsgren, Humble and Kim (2018) showed that organisations with a generative culture and loosely coupled architecture deliver faster and more reliably than those that centralise control (79, Forsgren et al., 2018), which argues for a central team that makes every use case easier to build rather than one that builds them all; AI Governance, Chapter 12 develops the organisational design.
Public-sector shared services
Public authorities face a sharper version of the problem. A national administration may comprise dozens of agencies, each a separate legal entity and deployer, each subject to Article 26(8) of the AI Act on registration of high-risk systems, and municipalities multiply the problem by hundreds. Sharing one platform, hosted by the existing Nordic shared-service bodies, is the only path that scales, and it must be multi-tenant by design, because each agency remains accountable to its own supervisory authority. Public programmes also inherit the procurement constraints that AI Governance, Chapter 7 describes: the rules of Directive 2014/24/EU on contract modification limit how far a supplier's offering may change during the term, so a model may have to be replaced through a new competition (9, European Union, 2014). The platform should assume that the model behind the gateway will change, and keep evaluation, prompts, and routing independent of any supplier.
Measuring industrialisation
A programme has escaped the pilot trap when its measures change from counts of pilots started to measures of flow, as Architecture as Code (Chapter 19) proposes for management as code: lead time from intake to production, the proportion of use cases on the golden path, the first-attempt pass rate at the readiness review, and the proportion of production systems with current evidence. Each is computable from the repository, so the steering committee can steer with evidence, the subject of AI Governance, Chapter 16.
Nordvik Region and Hansa Mutual
Hansa Mutual's claims triage assistant moved from pilot to production on the shared platform, and the same golden path carries it into further markets. Each new market is a new readiness review, covering language, national consumer rules, staff consultation, and, for the United Kingdom and Swiss entities, the national rules that replace the AI Act rows. The platform makes scaling cheap; the jurisdictions make it deliberate.
Nordvik Region's high-risk triage system was procured, so its readiness depends on the supplier's conformity documentation, the Region's own FRIA and EU database registration, and the regional executive committee. Its staff assistant (AIDR-0211) moved from a pilot of about 2,400 staff to about 3,200, the public-sector scale-out this chapter describes. Where Hansa risks scaling faster than each market's rules allow, Nordvik risks pilot purgatory: pilots waiting for budget or skills long after the evidence supports going live.
Summary
Pilots stall because they are built beside the estate rather than on it, and because the controls production demands are deferred until they become the most expensive thing to add. The remedy is a platform offered as a product, with golden paths that carry the controls, shared components for model access, evaluation, and guardrails, and an intake process that makes the decision to invest reviewable. The readiness review becomes a codified gate mapped to Article 26 of the AI Act, with Articles 4, 27, and 50 alongside, and a platform serving many agencies keeps their tenancy separate.
A platform, a gate, and a register serve many teams only when someone owns each use case, control, and exception, which AI Governance, Chapter 12 treats. This chapter closes Part V. In Part VI, Chapter 24 sets out the platform and security conclusions for Nordvik Region and Hansa Mutual, and Chapter 25 consolidates what it takes to build and secure AI platforms in European enterprises and public authorities.
Sources
- Paleyes, A., Urma, R.-G. & Lawrence, N. D. (2022). Challenges in Deploying Machine Learning: A Survey of Case Studies. ACM Computing Surveys, 55(6), Article 114. (67, Paleyes et al., 2022)
- Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F. & Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NeurIPS 2015). (43, Sculley et al., 2015)
- Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B. & Zimmermann, T. (2019). Software Engineering for Machine Learning: A Case Study. Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP '19), pp. 291–300. (65, Amershi et al., 2019)
- European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
- Nordqvist, G. (2025). Architecture as Code. The Transformed Architecture, Book 1, published at aac.geon.se. (72, Nordqvist, 2025)
- Skelton, M. & Pais, M. (2019). Team Topologies: Organizing Business and Technology Teams for Fast Flow. Portland, OR: IT Revolution Press. ISBN 978-1-942788-81-2. (78, Skelton & Pais, 2019)
- European Parliament and Council of the European Union. Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Official Journal of the European Union, 4 May 2016. (2, European Union, 2016)
- Breck, E., Cai, S., Nielsen, E., Salib, M. & Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Proceedings of the 2017 IEEE International Conference on Big Data, pp. 1123–1132. (66, Breck et al., 2017)
- Forsgren, N., Humble, J. & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. Portland, OR: IT Revolution Press. ISBN 978-1-942788-33-1. (79, Forsgren et al., 2018)
- European Parliament and Council of the European Union. Directive 2014/24/EU on public procurement. Official Journal of the European Union, 28 March 2014. (9, European Union, 2014)