What are reproducibility and governance in machine learning?

In this article

1. Before you start
2. Why interviewers ask this
3. Reproducibility and governance from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the learning path on taking a model to production and assumes you have trained a model before, as covered in machine learning algorithms and how they work.

Why interviewers ask this

A model that cannot be re-created or explained is a liability the moment someone outside the team asks about it. Candidates report prompts such as designing a pipeline so any training run can be reproduced months later, or answering a regulatory audit that wants data lineage, model cards, test results and decision logs for several production models within two weeks. Related reported prompts ask how you handle test set contamination and dependency drift.

A weak answer says "we use Git and a fixed seed." A strong answer separates the two ideas: what you pin so a run can be repeated, and what you record so someone else can check it. A Staff-level answer also says where repeatability ends, because a fixed seed does not make results identical across hardware or library releases, and names which obligations are law and which are voluntary.

Reproducibility and governance from first principles

Reproducibility

Reproducibility is the ability to re-create a training run and obtain a result that matches the original, within a stated tolerance, by reusing the same pinned code, data, parameters, environment and random seeds, so that a reviewer can check a result instead of trusting it.

Lineage record

A lineage record is a stored entry that links one model version to everything that produced it: the code version, a fingerprint of the data, the parameters, the environment image and the random seed. Any deployed version can then be traced back to its inputs without guessing.

Model card

A model card is a short document that travels with a trained model and reports its intended use, how it was evaluated across different conditions and groups, and its caveats, as proposed by Mitchell et al. (2019) for any released model.

In plain words

Think of a recipe card and a kitchen logbook. The recipe card lists the ingredients and steps. The logbook records which batch of flour you used, which oven, and who tasted the result. Reproducibility is the recipe card done carefully. Governance is the logbook plus the rules about who signs off.

The analogy breaks in one place. A kitchen result varies by taste, but a model run varies by library version, hardware and the order in which numbers are added. So the recipe must pin more than the ingredients. In one precise sentence, reproducibility is the technical ability to re-create a run, and governance is the policy, ownership, approval and evidence layer around it. To be exact, the same team rerunning with the same setup is called repeatability, so this page's sense of reproducibility sits close to it.

Step 1: Pin code, data and parameters

Pinning means writing down the exact version of every input, so the run can be repeated. Code goes in Git. Parameter files can live in Git too, but large datasets do not fit there well, so they need a pointer: a fingerprint committed in Git while the data sits in storage.

Pinned data is identified by its hash. Common cryptographic hashes make an accidental match practically impossible, while the 32-bit toy hash in the script behind this page is only illustrative. The Data Version Control (DVC) documentation describes a pipeline defined in dvc.yaml, with its state pinned in dvc.lock, which captures hashes of the dependencies and the values of the parameters that were used. Both files are versioned with Git, so one commit identifies the code, the data fingerprint and the parameters together.

Input: the training code, the dataset and the parameter file. Output: one commit that names all three. Nothing here yet pins the machine or the random numbers, which is the next step.

Step 2: Pin the environment and the randomness

The environment is the operating system, language, libraries and drivers a run executes in. Pin it with a locked dependency list and one container image built once and reused in development, testing and production, so the same bytes run everywhere.

Setting seeds is necessary, but it is not enough. The PyTorch documentation says completely reproducible results are not promised across releases, individual commits or different platforms, and that results may differ between central processing unit (CPU) and graphics processing unit (GPU) runs even with identical seeds. It also offers a switch that makes PyTorch use deterministic algorithms where they exist and raise an error where they do not, and it notes that deterministic choices can cost speed.

Input: a seed and a locked environment. Output: a run that repeats on the same platform and release. Record the platform too, as the next step does.

Step 3: Record lineage in a model registry

A lineage record is what turns pinned inputs into something an outsider can follow. The MLflow registry documentation says each registered model version is linked to the run, logged model or notebook that produced it, which enables full reproducibility. Each new model registered under the same name increments the version number, according to the versioned registry documentation.

A useful lineage record holds six fields in our toy example: a run identifier, the code version, the data fingerprint, the parameter fingerprint, the environment image and the seed.¹ Tags can carry gate results; the 2.14.0 registry page gives a tag named pre_deploy_checks as an example, set to a value that records the checks passed. The Databricks Workspace Model Registry, now marked legacy, records chronological model lineage (which run produced the model at a given time) and stage transitions, as of October 2026.

Input: a finished run. Output: a registered version that points back to it. Now any version in production can be traced, and the final step adds the evidence around it.

Step 4: Add governance evidence on top

Governance adds ownership, approvals and documents. A model card states intended use and evaluation results. Test results and decision logs show what was checked and who approved the release. This evidence sits on top of the lineage record rather than replacing it.

Law and frameworks decide how much you need. For high-risk systems, Article 11 of the European Union (EU) AI Act requires technical documentation drawn up before the system is placed on the market and kept up to date. Article 12 says high-risk systems shall technically allow for the automatic recording of events (logs). The National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF) is voluntary guidance organized into four functions, Govern, Map, Measure and Manage; it offers no certification, so say you follow or align with it rather than that you are "NIST compliant."

1 · Pin inputs2 · Control randomness3 · Record lineage4 · Add evidenceReplay

Step-by-step description

1. Code, data and parameters get pinned versions.
2. Environment and seeds are fixed and recorded.
3. The registry links each version to its run.
4. Cards, results and logs sit on top.

• Pinned inputs come first, so a run can be repeated.
• Each registered model version links to the run that produced it.
• Governance evidence sits on top of the lineage record.

Read the figure left to right: pinned inputs come first, so a run can be repeated. Each registered model version links to the run that produced it. Governance evidence sits on top of the lineage record.

Worked example

Take a toy audit: five production models, and four kinds of evidence an auditor asks for, namely lineage, test results, model cards and decision logs. The counts are illustrative, not a real company's state. In the toy state, lineage records exist for five of five models, test results for four, model cards for three and decision logs for one.¹

Text description of the diagram

• Lineage records exist for five of five models.
• Decision logs exist for one of five models.

Two small facts from the same script explain the rest. First, a fingerprint changes whenever its input changes: in the script, changing one learning-rate value changes the parameter fingerprint, which is why a pinned parameter file detects silent edits.¹ Second, the order of addition changes floating-point results: adding 0.1, 0.2 and 0.3 left to right gives 0.6000000000000001, while adding 0.2 and 0.3 first gives 0.6.¹ Parallel hardware adds numbers in varying order, which is one reason identical seeds still can differ across platforms.

What to do when the audit request arrives

An audit that asks for lineage, model cards, test results and decision logs for several production models within two weeks is answered by triage, not heroics. On day 1, inventory the production models and map each evidence kind against each model, as the toy table does: 13 of 20 items exist, so 7 need work. Pull lineage from the registry's run links. Backfill model cards from logged evaluation results.

Reconstruct decision logs from approval records such as pull-request reviews, tickets and registry transitions, and label each as reconstructed, with the date it was rebuilt, because a log written afterwards is not a contemporaneous one. Escalate to the owner and the auditor any gap that cannot be closed, such as a decision nobody recorded, rather than inventing it. Then automate capture, so the next audit is a query over stored records and not a project.

Trade-offs and edge cases

Reproducibility has limits and costs. The PyTorch documentation says exact repeats are not promised across releases or platforms, so a stated tolerance and a recorded platform are more honest than a claim of bit-for-bit identity. Deterministic settings can slow training, so decide where you need them: usually for audit-relevant runs and debugging, not every experiment. As of October 2026, recent MLflow versions favor aliases over stages for promotion, and Databricks marks its workspace registry legacy.

Text description of the diagram

• Reproducibility is the technical ability to re-create a run.
• Governance is the policy, ownership, approval and evidence layer.

Reproducibility is the technical ability to re-create a run. Governance is the policy, ownership, approval and evidence layer, and each fails differently.

Edge cases to name in an interview:

• Scope of law. The obligations in Articles 11 and 12 target high-risk systems, not every model.
• Cards are not audit evidence. A model card documents intended use and performance; technical documentation and event logs are separate obligations.

• Drift. A reproducible model can still decay in production; see why a model trained in January goes bad in July.
• Feature values. Pinned data is not enough if features are recomputed at serving; a feature store keeps training and serving values consistent.

How to answer it in an interview

The spoken structure: separate repeating a run from proving it, pin inputs, record lineage, admit the limits, then add the evidence layer.

Common mistakes

• "A fixed seed gives identical results." (wrong) The PyTorch documentation does not promise identical results across releases, platforms or CPU versus GPU. (right) "Seeds plus deterministic settings on a recorded platform give repeatable results within a tolerance."
• "Versioning code in Git is enough." (wrong) Data, parameters, environment and model artifacts need pinned versions too. (right) "Git holds the code, and a lock file holds the hashes of data and parameters."
• "We are NIST compliant." (wrong) The framework is voluntary guidance and offers no certification. (right) "We follow the framework's four functions, Govern, Map, Measure and Manage, and keep evidence for each."
• "Our model card is our audit evidence." (wrong) Cards document intended use and performance; logs and technical documentation are separate. (right) "The card is one document among lineage, test results and logs."

Follow-up questions

How do you handle test set contamination?

Fix the split before training, keep the test set out of every tuning decision, and store its fingerprint in the lineage record. Before reporting a score, check that no test example appears in the training data.

How do you manage dependency drift and environment consistency across development, testing and production?

Lock dependencies, build one container image once, and promote that same image through each stage instead of rebuilding it. PyTorch documents that results can differ across releases, so a changed library version should trigger a re-run of the checks, as in a continuous integration and delivery (continuous integration (CI)/continuous delivery (CD)) and continuous training design.

How would you design a centralized ML management platform?

Combine a pipeline tool that pins data and parameters, an artifact store, a model registry with lineage and approval states, and a documentation layer for cards and logs. The data platform under ML supplies the inputs it pins.

How do you operationalize model versioning and lineage?

Register every model under one name so the version number increments automatically, link each version to its run, and use tags to hold gate results such as pre-deploy checks. Block promotion when the lineage record is incomplete.

How would you design a secure ML data platform?

The general practice is least-privilege access, encryption of data in storage and in transit, and automatic access logs. The logs matter twice: security teams need access logs, and the same habit supports Article 12 style event recording, although access logs and operation-event logs are different records. Production monitoring, as in large language model (LLM) observability, can feed the same logs.

Is a model card enough for an audit?

No. A model card documents intended use, evaluation and caveats, while the EU rules for high-risk systems ask for technical documentation and event logs as well. Present the card together with lineage, test results and decision logs.

Sources

1. MLflow Model Registry — MLflow documentation, 26 Oct 2026
2. MLflow Model Registry (version 2.14.0 documentation) — MLflow documentation, 26 Oct 2026
3. Get Started: Data Pipelines — DVC documentation, 26 Oct 2026
4. Reproducibility (PyTorch notes on randomness) — PyTorch documentation, 26 Oct 2026
5. Model Cards for Model Reporting (arXiv 1810.03993) — arXiv (Mitchell et al.), 14 Jan 2019
6. EU AI Act, Article 11: Technical documentation — European Commission AI Act Service Desk, 26 Oct 2026
7. EU AI Act, Article 12: Record-keeping (Regulation (EU) 2024/1689) — European Commission AI Act Service Desk, 26 Oct 2026
8. Databricks Workspace Model Registry (legacy) — Databricks documentation, 11 Sep 2026
9. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 — National Institute of Standards and Technology, 26 Jan 2023