From notebook to service: a production-readiness checklist

In short: before a model serves real users, check seven things. The build can be repeated. The data is checked. The model beats a simple baseline on data it never saw. The service fails clearly. A release and a rollback are one command each. Someone is alerted when it drifts. And someone owns it.

Who this is for

The engineer who built the model, and the person who has to approve it for real use. Use it before the first release. Use it again when a service has run for a while and nobody has looked at it closely.

How to use it

Answer every check with yes, no or not needed. A no is not a failure. It is a risk you now know about. Next to each no, write who owns it and when it will be fixed, or why the risk is acceptable for now.

A scored list like this is not a new idea. A 2017 research paper, The ML Test Score, proposed a rubric of tests and monitoring needs for production machine learning. The checks below are our own: shorter, written for a first release, and in plain language. A copyable version is at the end.

01 The build can be repeated

If you cannot rebuild yesterday's model, you cannot tell whether today's model is better.

• Every dependency is pinned to an exact version.
• One command builds the environment from a clean checkout.
• Training runs from the repository, not from notebook cells run by hand in a chosen order.
• For each trained model, the code version, the data snapshot and the random seeds are recorded.

02 The data is checked

Many failures in production start in the data, not in the model. The 2015 paper Hidden Technical Debt in Machine Learning Systems describes why data dependencies cost more over time than code dependencies.

• Inputs are checked against a schema: names, types and allowed ranges.
• Missing values, duplicates and gaps are counted, and a limit stops the run when they jump.
• Training and serving compute features with the same code.
• You can name the data each model was trained on.

03 The model is measured honestly

A score on data the model has already seen tells you almost nothing about tomorrow.

• Quality is measured on data the model never saw. Where time matters, the test data comes after the training data.
• A simple baseline, such as a fixed rule or last week's value, is measured the same way, and the model beats it.
• The worst group of cases is reported next to the average.
• A test fails the build if quality drops below an agreed level.

04 The service behaves

Users meet the service, not the model. It has to fail in ways people can understand.

• Inputs and outputs have a written format.
• Bad input gets a clear error, not a guess.
• Every call has a time limit, and there is a fallback answer for when the model fails.
• Speed and memory are measured at the load you expect.

05 Releases are boring

A release that needs a careful afternoon will happen less often, and each one will carry more risk.

• Tests run on every change before it is merged.
• A release is one repeatable command.
• Going back to the last good version has been tried, not just planned.
• Each running model can be traced to its code and its data.

06 Someone will notice

A model can stop being right without throwing a single error. Monitoring is how you find out before your users do.

• Live inputs are compared with the training data, and a large shift raises an alert.
• The spread of outputs is tracked over time.
• Errors and slow responses raise alerts.
• Every alert goes to a named person, not to a shared inbox nobody reads.
• A person reviews a sample of live predictions on a schedule.

07 It has an owner

A service that only one person can run is one holiday away from an outage.

• A runbook says how to release, roll back and retrain.
• Retraining is a written procedure, not a memory.
• One person owns the service, and a second person can run it.
• There is a date to go through this checklist again.

What this checklist leaves out

Security review, privacy and legal review, and fairness testing. Each matters, and each needs its own specialist and its own checklist. This one covers engineering readiness only.

PRODUCTION-READINESS CHECKLIST
From notebook to service. Source: mattersettle.com/guides/notebook-to-service/
Mark each check: yes / no / not needed. For each no, write the owner and a date, or the reason it can wait.

01 THE BUILD CAN BE REPEATED
[ ] 1.1 Every dependency is pinned to an exact version
[ ] 1.2 One command builds the environment from a clean checkout
[ ] 1.3 Training runs from the repository, not from hand-run cells
[ ] 1.4 Code version, data snapshot and seeds are recorded per model

02 THE DATA IS CHECKED
[ ] 2.1 Inputs are checked against a schema (names, types, ranges)
[ ] 2.2 Missing values, duplicates and gaps are counted, with limits
[ ] 2.3 Training and serving compute features with the same code
[ ] 2.4 The training data of each model can be named

03 THE MODEL IS MEASURED HONESTLY
[ ] 3.1 Quality is measured on unseen data (later in time, if time matters)
[ ] 3.2 A simple baseline is measured the same way, and beaten
[ ] 3.3 The worst group of cases is reported next to the average
[ ] 3.4 A test fails the build if quality drops below an agreed level

04 THE SERVICE BEHAVES
[ ] 4.1 Inputs and outputs have a written format
[ ] 4.2 Bad input gets a clear error, not a guess
[ ] 4.3 Every call has a time limit, and there is a fallback answer
[ ] 4.4 Speed and memory are measured at the expected load

05 RELEASES ARE BORING
[ ] 5.1 Tests run on every change before merge
[ ] 5.2 A release is one repeatable command
[ ] 5.3 Rolling back to the last good version has been tried
[ ] 5.4 Each running model traces to its code and its data

06 SOMEONE WILL NOTICE
[ ] 6.1 Live inputs are compared with training data, with an alert
[ ] 6.2 The spread of outputs is tracked over time
[ ] 6.3 Errors and slow responses raise alerts
[ ] 6.4 Every alert goes to a named person
[ ] 6.5 A person reviews a sample of live predictions on a schedule

07 IT HAS AN OWNER
[ ] 7.1 A runbook covers release, rollback and retraining
[ ] 7.2 Retraining is a written procedure
[ ] 7.3 One owner, and a second person who can run it
[ ] 7.4 A date to go through this checklist again

Sources

• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. research paper, IEEE Big Data, 2017. Read 2026-10-16.
• Hidden Technical Debt in Machine Learning Systems. research paper, NIPS 2015. Read 2026-10-16.

Both are cited as background. The checks above are original and are not taken from either paper.

How this guide was written

Matter Settle publishes this guide without a byline. Its steps come from common engineering practice, not from any client project, and every example in it is invented. Outside work is cited above as background only, with the date it was read. The guide was last reviewed on 2026-10-16. If you find a mistake, tell us: the guide will be corrected and its review date changed.