Metrics in an AI contract: baseline, acceptance and rollback
A metric in an AI contract works only with a baseline, exact calculation, data source and acceptance procedure. The document should separate model quality from business outcome, assign ownership and define exceptions, monitoring and version changes. If a result cannot be reproduced or safely rolled back, a formal number protects neither party.
Key takeaways
- —Baseline and sample become part of acceptance.
- —The metric formula defines inclusion, exclusion and rounding.
- —A model score does not substitute for business outcome.
- —Model changes trigger an agreed retest.
- —Rollback and data ownership are fixed before launch.
The baseline records the starting system
Describe the current process, period, volume, task types, human decisions, errors and cost. Retain the source sample and labeling rules. Without a baseline, you cannot separate solution effect from seasonality, staffing or demand changes. A task-control service turns outcome criteria into observable states and ownership.
Metric definitions remove ambiguity
State observation unit, formula, source, window, exclusions, missing-data handling and rounding. Define an error and dispute owner. If human judgment is involved, attach a rubric, examples and reviewer-disagreement process. The AI implementation guide connects the metric to a process owner.
Contract schedule checklist
- Process objective and unacceptable outcome.
- Baseline, data version and sampling rules.
- Formula, source and metric owner.
- Acceptance set and independent verification.
- SLOs, monitoring, reporting and incident response.
- Rights to data, logs, models and derived assets.
- Change control, retest, rollback and access termination.
Acceptance separates model and business outcome
First verify technical quality on the agreed sample, then a limited process outcome under defined conditions. One model metric cannot guarantee business impact because implementation and user behavior also matter. NIST AI RMF helps structure measurement and risk management across the lifecycle.[1]
Model changes require retesting
Model version, prompt, knowledge base, tools and policy affect outcomes. The contract defines material changes, approval authority and which acceptance tests repeat. The public legal-agent case shows a domain legal workflow but does not replace adapting terms to jurisdiction and procurement context.
Limitations and failure modes
Risks include a metric without data access, acceptance on training samples, unilateral model changes and obligations for outcomes outside supplier control. A contract also does not replace technical controls. Rollback, data export and key revocation should be testable actions rather than generic wording.[1]
Frequently asked questions
Where should implementation start?
Start with a narrow task, baseline, process owner and safe manual fallback. Choose architecture only after defining data, actions and error consequences.
What metric is sufficient?
A sufficient metric is tied to an accepted useful outcome and has a source, formula, owner and exception rules.
When should automation stop?
Stop on unknown state, unapproved action, access violation, quality degradation or missing safe rollback.
Sources and evidence
- 1.AI Risk Management Framework — AI risk and accountability framework.
- 2.Guidelines for AI procurement — AI procurement, transparency and accountability context.
- 3.OECD AI Principles — Principles for robust, transparent and accountable AI.
- 4.Secure Software Development Framework — Change control and secure lifecycle.