An Introduction to Artificial Intelligence and Machine Learning

The first lecture in a series on AI and machine learning for clinicians. The goal is not to make you a data scientist, but to give you the tools to critique, review, and safely use the AI models that are already appearing in clinical practice.

Check Your Understanding

Answer these four questions to test the concepts from this lecture. Click “Show answer” to check your reasoning.

Question 1

Suppose you have two ML models, M1 and M2. M1 gets 90% accuracy on the training dataset and 90% on the validation dataset. M2 gets 95% on the training dataset and 85% on the validation dataset. Which model would you pick?

  • A. M2, as it performs better on the training dataset.
  • B. M1, as it performs better on the validation dataset.
  • C. Both are the same, as they have the same average accuracy across training and validation.
Show answer

Answer: B. A large gap between training and validation accuracy (M2) is a sign of overfitting. M1 performs equally well on data it has not seen, which is what matters for new patients.

Question 2

A vendor shows you a diagnostic AI model that reports 94% accuracy. Which check should matter most before you trust that number?

  • A. Confirm the model was trained on the largest dataset available.
  • B. Confirm the 94% is higher than the accuracy of competing models on the market.
  • C. Confirm the 94% was measured on patients like yours that the model never saw in training.
Show answer

Answer: C. An accuracy number only means something once you know it was measured on held-out data from a population like yours. A bigger dataset or a higher headline score than competitors does not tell you whether the model works on your patients.

Question 3

A sepsis model reaches 92% accuracy on Hospital A’s validation set, then drops to 70% when deployed unchanged at Hospital B, which has a different patient population and different equipment. What is the most likely reason?

  • A. The model learned signals specific to Hospital A that do not carry over to Hospital B.
  • B. The 22 point drop is normal run to run variation, so the two sites are really performing about the same.
  • C. The model will recover to 92% on its own as it sees more patients at Hospital B.
Show answer

Answer: A. A model validated at one site can still fail at another when it depends on signals specific to the first site, such as its patient mix or equipment. Option B is wrong because a 22 point drop is far larger than run to run noise. Option C is wrong because a deployed model does not keep learning from new patients unless it is retrained.

Question 4

A model is released that scores very highly at generating differential diagnoses on clinical vignettes published before 2024. On vignettes published from 2024 onward, its performance drops noticeably. What is the most likely reason?

  • A. The 2024 onward vignettes describe conditions too new for any model to diagnose.
  • B. Many of the pre-2024 vignettes were part of the model’s training data, while the 2024 onward vignettes were not.
  • C. The model can only handle textbook cases, not real clinical vignettes.
Show answer

Answer: B. When test cases have already appeared in a model’s training data, its score on them reflects memorization rather than diagnostic ability, which inflates the result. The 2024 onward vignettes were published after the model was trained, so it had never seen them, and its lower score there is a more honest estimate of how it actually performs. This is why a model should be evaluated only on data it could not have encountered during training. Option A is wrong because most newer vignettes describe ordinary presentations, not undiagnosable new conditions. Option C is contradicted by the model’s strong performance on the pre-2024 vignettes.

More Learning

If you want to build intuition for how a neural network actually works under the hood, this is a great next watch: