SW4 · TEACHER GUIDE
Facilitator’s Guide: Machine Learning Reliability Lab
Students work as a model testing team. They reuse an existing image classifier, collect evidence about its limits, and make a release decision for one named use.
Big question
How do we know whether a model works well enough for the people who will depend on it?
What students do
- Plan a fair 12-test suite.
- Stress-test the model and identify a repeated failure pattern.
- Audit what is well represented, weakly represented, or untested.
- Compare false alarms and misses for a named use.
- Complete a model card and recommend approving, revising, restricting, or rejecting the model for that use.
Teacher materials
Download the 32-slide teacher deck · Make a Google copy
Model Reliability Case File
Use the same case file all week. Choose the format that works best for you.
Download PDFDownload editable Word copy · Make a Google copy
Before Day 1
- Recover the saved Picture This model link. Open the Picture This Coding with Models assignment in SpeedGrader and check the student’s Day 4 submission comment. The Day 4 teaching deck asked students to paste the Teachable Machine share URL there immediately after Export Model → Upload.
- If a link is missing, rebuild one whole-class model in about 10 minutes. In Teachable Machine, start an Image Project with two simple labels such as pencil and eraser. Collect 20 to 30 webcam samples of each classroom object from more than one angle, train, then choose Export Model → Upload. Use that single share link for the class investigation. No student photos or personal images are needed.
- Decide whether students will complete the Word file digitally, annotate the PDF, or use printed copies.
- Test camera access on district Chromebooks if students will reopen their own models or help rebuild the class model.
Daily flow and leave-up screens
| Day | Teach | Student evidence | Deck |
|---|---|---|---|
| 1 | Demo vs. reliability; fair tests | 12-test plan | Slides 1 to 9; leave 9 up |
| 2 | Controlled testing; incident vs. pattern | Test log + claim | Slides 10 to 15; leave 15 up |
| 3 | Representation; targeted repair | Representation audit | Slides 16 to 20; leave 20 up |
| 4 | False alarms, misses, and labeling rules | Error-cost decision | Slides 21 to 26; leave 26 up |
| 5 | Model cards and release decisions | Complete case file + briefing | Slides 27 to 32; leave 31 up |
Watch for these misconceptions
- High confidence does not prove correctness. Students must compare the predicted label with the actual label.
- Untested does not mean failed. It means the team lacks evidence.
- More data is not automatically better data. Repairs should target the weak condition.
- No error type is always worse. The consequence depends on the intended use.
- Approval is not the only successful conclusion. Revise, restrict, and reject are valid when supported by evidence.
Shorter and flex routes
- Three-day route: combine Days 1 and 2, keep Day 3, then combine Days 4 and 5.
- Whole-class route: test one teacher model together and let teams audit different conditions.
- AI for Oceans: use it only as an optional Day 4 labeling/disagreement extension.
- Xello Make Plans: keep the existing Xello item as an optional/flex career-planning check-in; it is not required for the reliability investigation.
Bridge to AI Ethics
Close with: Reliable does not mean harmless. The next unit asks who benefits, who could be harmed, and who gets to decide whether the evidence is enough.
Teacher background: NIST AI Risks and Trustworthiness · Google ML: Accuracy, precision, and recall · Google ML: Dataset characteristics