Guides checklists and templates
Guides and templates
Original, dated and sourced guides for putting machine learning into production. Each has something you can copy into your own work today: a checklist, a list of questions, a template. Read them without signing up and without talking to anyone.
01 / The guides
Three guides, one artifact each
-
Checklist: From notebook to service: a production-readiness checklist
The checks a model should pass before it serves real users, grouped into seven areas: the build, the data, the model, the service, the release, the monitoring and the people who own it. Each check comes with a sentence on why it matters. For the engineer who built the model and the lead who has to sign off on it.
Published . Last reviewed .
-
Question list: Ten questions to ask any AI studio before you pay, including us
What to ask an outside team before you sign: what will exist at the end, who owns the code, who does the work, how quality will be measured, and how you can leave. For each question, what a useful answer sounds like and what should worry you. For founders, product leads and anyone buying AI work for the first time.
Published . Last reviewed .
-
Template: One-task LLM feature: build the evaluation set first
Why the evaluation set comes before the prompt, how to collect real examples, how to write a rubric a second person can apply, and how to use the set on every change. Includes an invented worked example. For developers and product teams building on a large language model.
Published . Last reviewed .
02 / Where to start
Pick the guide that matches your week
- You have a model in a notebook
- Start with the production-readiness checklist. Mark each check yes, no or not needed, and look hard at the noes.
- You are about to hire outside help
- Start with the ten questions. Ask every team the same ones, and write the answers down.
- You are building with an LLM
- Start with the evaluation set guide. Write the set before you change another prompt.
- Your problem is a data feed
- The checklist's data section is the place to begin, then the data pipelines page.
03 / How they are written
The rules every guide follows
A guide is only useful if you can trust it. These are the rules we hold each one to.
- One thing to copy. Every guide ends in a checklist, a question list or a template you can paste into your own notes.
- Invented examples, and labelled. Worked examples are made up, and they say so. None describes a client.
- Sources named. When a guide leans on outside work, it names the title, the publisher, the year and the date we read it.
- No borrowed numbers. Guides quote no results from other projects, and make no claims about past work.
- Dated. Each guide shows when it was published and when it was last reviewed.
- Corrected in the open. If you find a mistake, email us. The guide is fixed and the review date changes.
- No gate. No sign-up, no email form, no tracking.
04 / Words the guides use
A short glossary
The guides keep jargon to a minimum. These are the terms they cannot avoid.
- Baseline
- The simplest thing that could work, such as a fixed rule or last week's value. A model is only useful if it beats it.
- Held-out data
- Data kept aside and never used to build the model, so the test is fair.
- Evaluation set
- A fixed list of real examples with the answer you expect, used to score every version of a feature.
- Rubric
- A short list of criteria, each with a score and a sentence on what the score means.
- Quality gate
- A check that stops bad data or a weaker model before it goes any further.
- Rollback
- Going back to the last version that worked, with one command.
- Runbook
- Written steps for running and fixing a system, so it does not depend on one person's memory.
05 / Follow new guides
A feed, not a mailing list
New guides appear in an Atom feed at mattersettle.com/feed.xml. Add that address to any feed reader to see each new guide when it is published.
There is no mailing list, so there is nothing to sign up for and nothing to unsubscribe from. No email address is collected.
Next
Have a problem the guides do not cover?
Tell us what you are building in two or three sentences. An engineer replies.