Model Lifecycle Management: MLOps and LLMOps as Code

Chapter 6 made data a product with an owner, a contract, and lineage. This chapter follows that data into the model and describes the model lifecycle as a pipeline with gates rather than a sequence of projects: experiment tracking, the registry, promotion gates, deployment, rollback, and retirement; the ML Test Score as a readiness measure; what changes when the managed artefact is a prompt, an evaluation set, or an adapter; and how regulated organisations enforce separation of duties inside the pipeline. It extends Architecture as Code (Chapters 5 and 13) to the model (72, Nordqvist, 2025).

Figure 7.1: Every model version passes the same six promotion gates on its way to production, and monitoring evidence drives retraining, rollback or retirement.

Why the lifecycle needs its own discipline

Sculley and colleagues gave the field its vocabulary for what goes wrong when models are treated as ordinary software (43, Sculley et al., 2015). Their catalogue of hidden technical debt includes entanglement ("changing anything changes everything"), data dependencies that change without a commit, feedback loops in which a model influences its future training data, pipeline jungles of glue code, and dead experimental code paths. These debts accrue between training runs, so they are lifecycle problems.

Amershi and colleagues, studying machine learning teams at Microsoft, found three ways in which the workflow differs from software engineering: managing data is harder than managing code, model customisation demands skills most software teams lack, and models resist modular design (65, Amershi et al., 2019). Machine Learning Operations (MLOps) is the response. Kreuzberger, Kühl, and Hirschl derive nine principles for it, among them CI/CD automation, reproducibility, versioning of data, model, and code, continuous training and evaluation, metadata tracking, and continuous monitoring (68, Kreuzberger et al., 2023), and Google's practitioner guide describes the same processes supported by a model registry and metadata tracking (81, Google Cloud, 2021). What the literature leaves to the architect is wiring the components into a pipeline that an auditor can read and a platform team can run at scale.

The stages of the pipeline

Experiment tracking

Every training run records parameters, metrics, code version, data version, environment, and artefacts automatically. MLflow ties each run to the Git commit and data reference that produced it (37, MLflow, n.d.), and Kubeflow Pipelines offers the same capture inside a Kubernetes-native orchestration layer (38, Kubeflow, n.d.). The rule is that a model not traceable to its run does not enter the registry.

The model registry

The registry is to models what a container registry is to images: an immutable, signed artefact store with metadata. Each entry links to the tracking run, the model card from Chapter 5, the data product versions from Chapter 6, and the evaluation report, and promotion gates move versions between stages such as candidate, champion, and challenger. In a regulated organisation the registry also answers "which model version made this decision", so its history is retained for as long as those decisions can be challenged.

Worked example: a model registry entry

Illustrative example. The entry below records version 4.0.0 of the claims ranker in Nordvik Region's benefits-triage system, accepted in AIDR-0018 after its predecessor breached a fairness threshold (Chapter 5). The provider, TriageSoft (fictional), trains the model inside Nordvik's own platform, so the registry, the evidence and the approvals stay with the authority; pipelines write everything except the approvals.

# registry://claims-ranker/4.0.0 - Nordvik Region (illustrative)
model: claims-ranker
version: 4.0.0
artefact:
  uri: s3://nordvik-models/claims-ranker/4.0.0/model.ubj
  signature: sigstore://nordvik/claims-ranker/4.0.0
provenance:
  training_run: mlflow://nordvik/runs/8c41f0
  training_manifest: models/claims-ranker/training_run.yaml@a91c3e2
  data: [{product: claims-register, version: 2026-12}]
documentation:
  decision_record: AIDR-0018
  model_card: models/claims-ranker/4.0.0/model_card.md
ai_act:
  roles: [deployer]
  provider: TriageSoft (fictional)
  classification: high-risk
  change_envelope: models/claims-ranker/change_envelope.yaml
evaluation:
  precision_at_100: 0.82
  demographic_parity_difference: 0.02
  ml_test_score: 4.5
stages:
  - {stage: shadow, at: 2027-01-29T16:02Z, approved_by: model-risk/a.lind}
  - {stage: champion, at: 2027-02-10T07:30Z, approved_by: head-of-case-management}
previous_champions_warm: [3.2.0, 3.1.1]
retention:
  keep_until: last case handled with this version plus 10 years

The approved_by fields name people who did not build the model, which the four-eyes gate enforces. The retention line ties the registry to administrative law rather than storage cost: as long as a case handled with version 4.0.0 can be appealed, the authority must be able to show what that version was.

Promotion gates

A promotion gate is a set of automated checks that a model version must pass to move between stages. We insist on six:

1. Data gate. The training data satisfies its contract, lineage is recorded, and the lawful basis permits training for the stated purpose.
2. Reproducibility gate. A second run from the same code, data, and seed lands within a declared tolerance of the first.
3. Evaluation gate. The model meets every threshold in the AI Decision Record on the frozen evaluation set, including fairness metrics.
4. Comparison gate. The challenger beats the champion by a margin, or the record explains why not.
5. Readiness gate. The ML Test Score meets the minimum for the system's risk class.
6. Approval gate. For high-risk systems, a named approver who did not build the model confirms promotion.

Only the last gate involves a person; the others run in minutes inside the pull request, reading thresholds from the decision record so that record and gate cannot disagree. Appendix A contains a complete workflow that implements them.

Deployment strategies

Three strategies reduce the blast radius when probabilistic behaviour meets live traffic. Shadow deployment sends a copy of production traffic to the new version and logs its outputs without returning them, showing how the model would have behaved at the cost of double inference. Canary deployment serves the new version to one to five per cent of traffic before widening; because labels often arrive late, it relies on proxies such as prediction and confidence distributions and agreement with the champion. Champion–challenger keeps versions on recorded split traffic long enough to compare real outcomes, the only strategy that measures what the business cares about.

Rollback and retirement

Rollback is a registry operation, not a redeployment: moving the champion alias back is immediate, provided the previous version's environment, features, and dependencies still exist. We therefore keep at least two previous champions warm for any high-risk system.

Retirement is the stage most programmes forget. It follows the AI Decision Record's retirement criteria, the end of the purpose, or the end of a successor's overlap period, and it removes the endpoint, archives the registry entry, and records the date. However, logs that a high-risk system generated under Article 12 of the AI Act remain subject to retention obligations, of at least six months for providers under Article 19 and for deployers under Article 26(6), and Article 18 requires the provider to keep the technical and quality management documentation for ten years; retiring the model does not retire its logs (1, European Union, 2024).

Reproducibility

Reproducibility lets a model be regenerated from its inputs and lets an auditor confirm that the model in production is the one that was evaluated. It depends on pinning code by commit, data by immutable version or hash, environment by image digest, configuration including seeds, and hardware class, because results on different accelerators can differ at the margin. The training manifest codifies the pinning and is stored with the model.

training_run:
  model: claims-ranker
  code:
    commit: 7d3f2a1c9e4b5d6a8f0e1c2b3a4d5e6f7a8b9c0d
  data:
    - {product: claims-register, version: 2025-12}
  environment:
    image: registry.example.org/ml/train-py312@sha256:9b8c7d6e5f4a
    accelerator: nvidia-a100-80gb
  config:
    seed: 20260302
    hyperparameters: {learning_rate: 0.05, max_depth: 8, n_estimators: 600}
  reproducibility_tolerance:
    precision_at_100: 0.005

The tolerance allows for non-deterministic parallel hardware; a model outside it has an untrustworthy evaluation.

The ML Test Score as a readiness gate

Breck and colleagues at Google proposed the ML Test Score as a rubric for production readiness: twenty-eight tests in four areas, features and data, model development, infrastructure, and monitoring (66, Breck et al., 2017). Each test scores half a point when performed manually and documented and a full point when automated, and the overall score is the minimum across the four areas, so that exhaustive data tests cannot compensate for untested monitoring. Many of the tests, such as reproducible training and rollback, are the gates above.

We set the minimum by risk class. The class comes from the classification that AI Governance, Chapter 8 codifies and records in the AI Decision Record; the pipeline reads the tier from the record, so a reclassification raises the minimum without anyone editing the pipeline. A minimal-risk internal system may ship at a lower score with its gaps recorded. A high-risk system does not promote below a score that implies automated tests in every area, because Article 9 of the AI Act requires testing against prior defined metrics throughout the development process and in any event before the system is placed on the market, and Article 15 requires accuracy and robustness that only running tests can evidence (1, European Union, 2024).

LLMOps: what changes with foundation models

When the model is a general-purpose AI model obtained from a third party, the organisation neither trains it nor can usually inspect it. The lifecycle still exists but manages different artefacts, a practice called LLMOps.

Lifecycle concern	Classical MLOps artefact	LLMOps artefact
The thing being versioned	Trained model weights	Prompt templates, retrieval and guardrail configuration, fine-tuned adapters
Evaluation set	Held-out labelled data	Curated prompts with reference answers and rubrics
Metric	Accuracy, precision, parity	Faithfulness, task success, refusal rate, cost per task
Promotion unit	Model version	Compound system: model plus prompt plus configuration
Drift	Data and concept drift	Vendor model updates, corpus changes, prompt regressions

Prompts are configuration, and configuration is code: a prompt template is versioned, reviewed in a pull request, and evaluated before promotion, as Chapter 10 develops. Evaluation sets are the labelled data of LLMOps and need an owner, a version, and a change process; because a general-purpose model can regress on one task when a vendor improves another, the set runs on a schedule and whenever the vendor announces an update.

Fine-tuning changes the registry, not the principle. Low-Rank Adaptation (LoRA), proposed by Hu and colleagues, freezes the pre-trained weights and trains small rank-decomposition matrices, reducing trainable parameters by orders of magnitude and allowing adapters to be swapped without redeploying the base model (61, Hu et al., 2022). An adapter is therefore a model version with a declared dependency on a base model version, with its own card, manifest, and evaluation, and it is re-evaluated whenever the base model changes.

Worked example: a promotion gate for a compound system

Illustrative example. In LLMOps the gate names every component version and the thresholds the combination must meet. Hansa Mutual's gate for the claims assistant recorded in AIDR-0021 takes its thresholds from the decision record, not the pipeline code.

# gates/claims-assistant.yaml - Hansa Mutual (illustrative)
kind: PromotionGate
metadata:
  system: claims-assistant
  decisionRecord: AIDR-0021
candidate:
  model: registry://gpai-large-eu/2026-04
  systemPrompt: prompts/claims-assistant/system@v4
  retrievalConfig: rag/claims-assistant.yaml@v9
  index: policy-wordings-v12
  guardrails: guardrails/claims-assistant@v2
evaluation:
  suite: eval/claims-assistant/v6
  thresholds:
    faithfulness: {min: 0.95}
    citation_accuracy: {min: 0.98}
    correct_abstention_out_of_scope: {min: 0.90}
    harmful_content_rate: {max: 0.0}
    cost_eur_per_answer: {max: 0.04}
  regression:
    maxDropAnyMetric: 0.02
    slices: [motor, home, commercial-property, travel, de, sv]
triggers: [change to any candidate component, provider model version notice, weekly schedule]
approval:
  approverRole: model-risk-management
  excludeAuthorsOfChange: true

The slices list prevents an aggregate pass from hiding a regression in one product line or language, and the provider notice trigger treats a vendor update as a change to Hansa Mutual's system, which is how the gate implements the DORA change-management controls discussed below.

Separation of duties in regulated organisations

A pipeline that one person can push a model through end to end is not acceptable in a bank, an insurer, a hospital, or a public authority. Three instruments shape the design.

DORA, Regulation (EU) 2022/2554, requires financial entities under Article 9 to implement documented, risk-based controls for ICT change management (3, European Union, 2022). A model promotion is an ICT change and the gates are the controls; the approval gate enforces the four-eyes principle by rejecting any approver who appears in the training run's commit history. Article 17 of the AI Act requires providers of high-risk systems to operate a quality management system covering design, quality control, and test and validation procedures (1, European Union, 2024); the versioned pipeline definition is the most honest description of those procedures, because it is the one that executes.

Article 43(4) of the AI Act requires a new conformity assessment when a high-risk system is substantially modified, but for systems that continue to learn after being placed on the market, changes the provider pre-determined at the initial assessment and documented in the technical documentation do not count as substantial modifications (1, European Union, 2024). The implication is large: a system retrained quarterly declares its retraining procedure, thresholds, and acceptable ranges of change up front, so each retraining within that envelope is a planned change through the gates rather than a new product returning to the notified body.

Worked example: a predetermined change envelope

Illustrative example. The provider retrains Nordvik Region's claims ranker quarterly inside the authority's platform. The envelope, referenced from the technical documentation and the registry entry, separates planned retraining from changes that need full reassessment.

# models/claims-ranker/change_envelope.yaml - Nordvik Region (illustrative)
envelope:
  system: claims-prioritisation
  model: claims-ranker
  approved_in: AIDR-0018
  planned_changes:
    retraining:
      interval_days: 90
      data: same data product, newer release, same feature list
      hyperparameter_ranges: {max_depth: [6, 10], n_estimators: [400, 900]}
  acceptance:
    precision_at_100: {min: 0.80}
    demographic_parity_difference: {max: 0.05}
  outside_envelope:
    - new or removed features
    - change of algorithm family
    - change of intended purpose or user group
    - acceptance criterion breached after retraining
  outside_envelope_action: new decision record and assessment before deployment

The outside_envelope list matters as much as the planned changes. Adding a feature is the change teams are most tempted to treat as routine, and listing it means the pipeline, rather than a busy engineer, decides that it needs a new record and a fresh assessment under Article 43(4). For a public authority the same pattern serves administrative law, because an officer with authority to accept the version promotes it and the registry shows which version handled any citizen's case.

Caseworkers also receive a plain-language release note; the note for version 4.0.0 reports the fall in precision at 100 from 0.83 to 0.82 alongside the improvement in parity from 0.06 to 0.02, so that staff who notice a change in the queue find it already explained.

Practical consideration: when the provider trains the model

Many organisations, and most public authorities, do not train the models they deploy, so the question becomes whose pipeline the gates live in. The answer that preserves the lifecycle is that the deploying organisation owns the gates and the registry, and the provider either trains inside the organisation's platform or delivers what the gates consume: a training manifest, an evaluation report on the organisation's frozen evaluation set, a model card, and the data lineage. AI Governance, Chapter 7 treats procurement; from the lifecycle's point of view the tender names these artefacts, the change envelope, and the right to run the organisation's own evaluation, because a public authority bound by procurement law cannot add such terms after award without risking a substantial modification of the contract under Article 72 of Directive 2014/24/EU (9, European Union, 2014).

Across the usage modes

The lifecycle of this chapter is complete only where the organisation builds the system. Across the usage modes of Chapter 2 (usage-mode matrix), what it versions shrinks as the supplier's share grows.

Mode	What the organisation versions	Lifecycle owner
Mode 0 – Ambient AI	Nothing internally; the inventory records the feature and supplier version	Supplier
Mode 1 – Conversational assistants	The approved service's settings; shared prompts	Supplier for the model; organisation for settings
Mode 2 – Embedded workplace assistants	Tenant configuration, grounding scope, custom agents	Shared
Mode 3 – Agentic work tools	Skills, agent configurations, instruction files, permissions	Organisation, as Chapter 8 describes
Mode 4 – Built AI applications	Data, models, prompts, retrieval configuration, evaluation sets	Organisation
Mode 5 – Autonomous production agents	As Mode 4, plus manifests, approval policies, agent evaluation gates	Organisation

In Modes 1 and 2 a vendor model update cannot be pinned, so the organisation keeps an evaluation set for important tasks and runs it whenever the vendor announces a change.

Nordvik Region and Hansa Mutual

For Nordvik Region the lifecycle is shared with its provider. TriageSoft retrains the claims ranker inside the authority's platform within the change envelope declared under Article 43(4), while Nordvik holds the registry, the gates, the release note and the evidence (1, European Union, 2024). Its decisive control is acceptance by an official with the authority to accept a version, which matters when a resident's decision is appealed.

Hansa Mutual, by contrast, runs the lifecycle itself, with the compound system as the unit of promotion. Its separation of duties follows DORA's change-management requirements for the EU entities (3, European Union, 2022) and the governance requirements of Solvency II (239, European Union, 2009), which the European insurance supervisor's opinion on AI governance links to AI systems (240, EIOPA, 2025). The UK and Swiss entities apply the same gates as group policy, and because the gate slices by language and product line, a release is promoted market by market rather than everywhere at once.

Summary

The model lifecycle is a pipeline with gates because a sequence of handovers is how hidden technical debt accrues. Tracking, an immutable registry, and automated gates come before any human approval; rollback works only while previous versions stay deployable, and retirement does not end log retention.

LLMOps keeps the lifecycle but versions prompts, evaluation sets, guardrails, and adapters, treating a vendor update as a change to the compound system. In regulated organisations DORA's change management, the AI Act's quality management system, and the change envelope of Article 43(4) are implemented most faithfully by the pipeline itself. Chapter 8 widens the frame from the model to the whole delivery loop around it.

Sources

1. Nordqvist, G. (2025). Architecture as Code. The Transformed Architecture, Book 1, published at aac.geon.se. (72, Nordqvist, 2025)
2. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F. & Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NeurIPS 2015). (43, Sculley et al., 2015)
3. Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B. & Zimmermann, T. (2019). Software Engineering for Machine Learning: A Case Study. Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP '19), pp. 291–300. (65, Amershi et al., 2019)
4. Kreuzberger, D., Kühl, N. & Hirschl, S. (2023). Machine Learning Operations (MLOps): Overview, Definition, and Architecture. IEEE Access, 11, pp. 31866–31879. (68, Kreuzberger et al., 2023)
5. Google Cloud. Practitioners guide to MLOps: A framework for continuous delivery and automation of machine learning. White paper, May 2021. (81, Google Cloud, 2021)
6. MLflow Project. MLflow Documentation. LF Projects, LLC. (37, MLflow, n.d.)
7. Kubeflow Authors. Kubeflow Documentation. Cloud Native Computing Foundation. (38, Kubeflow, n.d.)
8. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
9. Breck, E., Cai, S., Nielsen, E., Salib, M. & Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Proceedings of the 2017 IEEE International Conference on Big Data, pp. 1123–1132. (66, Breck et al., 2017)
10. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. & Chen, W. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR 2022). (61, Hu et al., 2022)
11. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). Official Journal of the European Union, 27 December 2022. (3, European Union, 2022)
12. European Parliament and Council of the European Union (2009). Directive 2009/138/EC on the taking-up and pursuit of the business of Insurance and Reinsurance (Solvency II) (recast). Official Journal of the European Union, L 335, 17 December 2009. (239, European Union, 2009)
13. European Insurance and Occupational Pensions Authority (2025). Opinion on Artificial Intelligence governance and risk management. EIOPA, 6 August 2025. (240, EIOPA, 2025)
14. European Parliament and Council of the European Union. Directive 2014/24/EU on public procurement. Official Journal of the European Union, 28 March 2014. (9, European Union, 2014)