Machine Learning Foundations

Jul 6, 2026

The first lecture in a series on AI and machine learning for clinicians. The goal is not to make you a data scientist, but to give you the tools to critique, review, and safely use the AI models that are already appearing in clinical practice. This first lecture focuses on the fundamentals of classical machine learning. Large language models (LLMs) and generative AI are not covered here; they come later in the series.

What this lecture covers

A ground-up tour of how machine-learning models work and how to judge them, written for clinicians. No math background needed. We discuss:

What ML actually is. Machine learning means learning patterns from data to predict new cases, rather than following rules someone wrote by hand.
Classes and types of problems. The three ways models learn (supervised, unsupervised, reinforcement) and what a model actually outputs: a number (regression), a category (classification), the location of something (detection/segmentation), or new content (generation). The same task can run on tables, images, text, or signals. Most clinical tools are supervised classification or regression.
How a model gets built: the lifecycle. The stages every model passes through: define the problem, collect data, label it, split into train/validation/test, train, evaluate, pick a threshold, deploy, monitor.
A simple example of gradient descent: recovering BMI. Fitting height and weight to BMI by trial and error (gradient descent). Because BMI is a formula, the model recovers it exactly, a clean way to watch the mechanism work. We can see how the model minimizes a loss function.
A harder example: predicting complications. The same patients, but a noisy, non-linear outcome a straight line can’t capture. We let the line bend (polynomial regression) and swap in other model types (decision tree, SVM). You can determine which ML model based on the complexity of data at hand.
Neural networks without the mystique. A “perceptron” is just a weighted sum passed through an activation function. Stack layers and the network learns its own features straight from raw pixels, which is the core win of deep learning.
Overfitting and generalization. A model can score 100% on its training data and still be useless. Why you split data into train / validation / test, why peeking at the test set is the top reason papers don’t replicate, and the splits that matter clinically: by patient, over time, and at an external site.
What the performance numbers mean. You already know sensitivity, specificity, PPV, NPV, ROC, and AUC from diagnostic testing. ML just renames some of them: precision = PPV, recall = sensitivity, plus F1 and accuracy.
From prediction to decision. A model outputs a score; you choose the cutoff. Move the threshold and the same model either catches everyone or misses real cases, very different clinical behavior from one model.
When models fail in medicine. Three real cases: an algorithm that used cost as a stand-in for illness and under-referred Black patients (labels are often proxies); a pneumonia detector that learned which hospital an X-ray came from instead of the disease (right answer, wrong reason); and silent performance drift when a model meets a new population, with no alarm to warn you.
Trust, takeaways, and a checklist. Does the model decide, or just flag for a human? Closes on five durable takeaways and a six-question checklist for any AI claim.

Five takeaways

A model that performs well on training data may still fail on new patients.
The right metric depends on the clinical use; accuracy alone misleads.
A model can be right for the wrong reason.
Before trusting a model, ask where the data came from, how it was validated, and whether it matches your setting.
A model may be perfect on paper, but how it’s deployed drives its real-world effectiveness.

Six questions for any AI claim

Data / label: what did it learn from, and is the label the thing you care about or a proxy?
Validation: was it tested on truly new data (different patients, site, or era)?
Metrics: at what threshold? Is calibration shown? Performance by subgroup?
Comparator: better than what? Clinician judgment, an existing score, or usual care?
Action: does a good prediction actually change management?
Monitoring: how will you know when it starts failing?

Check Your Understanding

Question 1

Suppose you have two ML models, M1 and M2. M1 gets 90% accuracy on the training dataset and 90% on the validation dataset. M2 gets 95% on the training dataset and 85% on the validation dataset. Which model would you pick?

A. M2, as it performs better on the training dataset.
B. M1, as it performs better on the validation dataset.
C. Both are the same, as they have the same average accuracy across training and validation.

Show answer

Answer: B. A large gap between training and validation accuracy (M2) is a sign of overfitting. M1 performs equally well on data it has not seen, which is what matters for new patients.

Question 2

A vendor shows you a diagnostic AI model that reports 94% accuracy. Which check should matter most before you trust that number?

A. Confirm the model was trained on the largest dataset available.
B. Confirm the 94% is higher than the accuracy of competing models on the market.
C. Confirm the 94% was measured on patients like yours that the model never saw in training.

Show answer

Answer: C. An accuracy number only means something once you know it was measured on held-out data from a population like yours. A bigger dataset or a higher headline score than competitors does not tell you whether the model works on your patients.

Question 3

A sepsis model reaches 92% accuracy on Hospital A’s validation set, then drops to 70% when deployed unchanged at Hospital B, which has a different patient population and different equipment. What is the most likely reason?

A. The model learned signals specific to Hospital A that do not carry over to Hospital B.
B. The 22 point drop is normal run to run variation, so the two sites are really performing about the same.
C. The model will recover to 92% on its own as it sees more patients at Hospital B.

Show answer

Answer: A. A model validated at one site can still fail at another when it depends on signals specific to the first site, such as its patient mix or equipment. Option B is wrong because a 22 point drop is far larger than run to run noise. Option C is wrong because a deployed model does not keep learning from new patients unless it is retrained.

Question 4

A model is released that scores very highly at generating differential diagnoses on clinical vignettes published before 2024. On vignettes published from 2024 onward, its performance drops noticeably. What is the most likely reason?

A. The 2024 onward vignettes describe conditions too new for any model to diagnose.
B. Many of the pre-2024 vignettes were part of the model’s training data, while the 2024 onward vignettes were not.
C. The model can only handle textbook cases, not real clinical vignettes.

Show answer

Answer: B. When test cases have already appeared in a model’s training data, its score on them reflects memorization rather than diagnostic ability, which inflates the result. The 2024 onward vignettes were published after the model was trained, so it had never seen them, and its lower score there is a more honest estimate of how it actually performs. This is why a model should be evaluated only on data it could not have encountered during training. Option A is wrong because most newer vignettes describe ordinary presentations, not undiagnosable new conditions. Option C is contradicted by the model’s strong performance on the pre-2024 vignettes.

Question 5

A team builds a model to flag pneumonia on chest X-rays by showing it thousands of images already labeled “pneumonia” or “no pneumonia.” In what sense did the model “learn”?

A. A radiologist wrote explicit rules for pneumonia that the model now applies.
B. The model memorized a reference textbook and looks each new X-ray up against it.
C. The model found patterns that separate the labeled examples, without being given explicit rules.

Show answer

Answer: C. This is supervised learning: shown many labeled examples, the model adjusts itself to find patterns that separate “pneumonia” from “no pneumonia,” rather than following rules a human wrote out (A) or looking each case up in a stored reference (B). The labels are what it learns from.

Question 6

A sepsis model outputs a risk score from 0 to 1, and the team picks a threshold above which it raises an alert. They lower that threshold from 0.5 to 0.2. What is the expected effect?

A. The model becomes more accurate at every threshold, because the score itself has improved.
B. It raises more alerts, catching more true cases but also producing more false alarms.
C. It raises fewer alerts, missing more true cases but producing fewer false alarms.

Show answer

Answer: B. Lowering the threshold flags more patients, so sensitivity rises (more true cases caught) at the cost of more false positives. The underlying score is unchanged, so nothing about the model “improved” (A), and lowering the cutoff raises rather than lowers the number of alerts (C describes raising it). One model gives very different clinical behavior depending on where the cutoff sits.

Machine Learning Foundations

Check Your Understanding

Question 1

Question 2

Question 3

Question 4

Question 5

Question 6

Further reading