Alright, so this is Part One of an intro to AI and machine learning, and it's really meant to be a primer for clinicians. The goal isn't to turn you into a data scientist; it's really to give us the tools to critique, review, and even use the AI models that are being deployed in clinical practice today. So the three things we'll cover are, one, what AI and machine learning actually is; two, how do machines "learn," and what does that really mean; and three, how do you judge whether or not you should trust an AI model, and what are some things that can go wrong along the way.
Alright, so this is Part One of an intro to AI and machine learning, and it's really meant to be a primer for clinicians. The goal isn't to turn you into a data scientist; it's really to give us the tools to critique, review, and even use the AI models that are being deployed in clinical practice today. So the three things we'll cover are, one, what AI and machine learning actually is; two, how do machines "learn," and what does that really mean; and three, how do you judge whether or not you should trust an AI model, and what are some things that can go wrong along the way.
So let's take a step back. AI is the umbrella term, and it can mean a lot of different things: expert systems, hard-coded rules, general intelligence, all of it. Machine learning is a subset of AI, and the general idea is that a machine learning model can learn a pattern from data and then apply that pattern to new data it hasn't seen before. This is a little bit different from the hard-coded rules we're normally used to. Think of something like CHA₂DS₂-VASc, that's a classical algorithm, right? Strict "if the patient has this, then do that." Machine learning models are less rigid; they make predictions based on the patterns they've learned.
So let's take a step back. AI is the umbrella term, and it can mean a lot of different things: expert systems, hard-coded rules, general intelligence, all of it. Machine learning is a subset of AI, and the general idea is that a machine learning model can learn a pattern from data and then apply that pattern to new data it hasn't seen before. This is a little bit different from the hard-coded rules we're normally used to. Think of something like CHA₂DS₂-VASc, that's a classical algorithm, right? Strict "if the patient has this, then do that." Machine learning models are less rigid; they make predictions based on the patterns they've learned.
Now, even deeper within machine learning, there's something called deep learning, and that's what a lot of people are familiar with: neural networks, image recognition, FaceID, AlphaGo, all of that fits in here. And then large language models like ChatGPT, Claude, Gemini, those are a specific type of neural network built on a certain architecture that does text generation really impressively.
Now, even deeper within machine learning, there's something called deep learning, and that's what a lot of people are familiar with: neural networks, image recognition, FaceID, AlphaGo, all of that fits in here. And then large language models like ChatGPT, Claude, Gemini, those are a specific type of neural network built on a certain architecture that does text generation really impressively.
So when we talk about machine learning, there's sort of a few big classes, and these aren't the only ones, but it's helpful to frame things this way. The first is supervised learning: you have patient data plus some labeled output, like readmission within 30 days, and you feed all of that into a model so it can learn the right answer for a given input. Unsupervised learning is different: there's no specific label. The model has to figure out on its own which patients are most similar to each other. If you think about Spotify recommending songs, that's a form of unsupervised learning. And then there's reinforcement learning, where the model takes actions in some environment and gets rewarded or penalized. You might have seen those YouTube videos of an AI learning to play Mario; that's reinforcement learning.
So when we talk about machine learning, there's sort of a few big classes, and these aren't the only ones, but it's helpful to frame things this way. The first is supervised learning: you have patient data plus some labeled output, like readmission within 30 days, and you feed all of that into a model so it can learn the right answer for a given input. Unsupervised learning is different: there's no specific label. The model has to figure out on its own which patients are most similar to each other. If you think about Spotify recommending songs, that's a form of unsupervised learning. And then there's reinforcement learning, where the model takes actions in some environment and gets rewarded or penalized. You might have seen those YouTube videos of an AI learning to play Mario; that's reinforcement learning.
Once you've picked the class, there are a few main task types these models can actually do. A regression task predicts a number: like given a patient's last five glucose readings, can we predict the sixth. A classification task predicts a category: will this patient develop sepsis, is this mass benign or malignant. A detection or segmentation task locates something in the data: like, given this chest x-ray, can you draw a box around the lung nodule. And then generation tasks, which have gotten really popular recently: given some input, can you generate an output, like drafting a discharge summary.
Once you've picked the class, there are a few main task types these models can actually do. A regression task predicts a number: like given a patient's last five glucose readings, can we predict the sixth. A classification task predicts a category: will this patient develop sepsis, is this mass benign or malignant. A detection or segmentation task locates something in the data: like, given this chest x-ray, can you draw a box around the lung nodule. And then generation tasks, which have gotten really popular recently: given some input, can you generate an output, like drafting a discharge summary.
And, you know, we think a lot about text data, but this really applies to any type of data: tabular data like labs, imaging data like x-rays, text data like clinical notes, or signal data like ECGs. The task is the same; only the type of data going into the model changes.
And, you know, we think a lot about text data, but this really applies to any type of data: tabular data like labs, imaging data like x-rays, text data like clinical notes, or signal data like ECGs. The task is the same; only the type of data going into the model changes.
The simplest example I can think of is linear regression. Now, linear regression is something you can do without machine learning at all, but I find it really helpful as a visual for how these models work. So let's take a sort of contrived example. Let's say we've forgotten what the formula for BMI is, and we have a bunch of patients with a height and a weight, and we have their BMI, but we don't actually know the formula. Can we build a model that recovers the relationship from the data? This is a supervised task because we have labels, and it's a regression task because we're predicting a number.
The simplest example I can think of is linear regression. Now, linear regression is something you can do without machine learning at all, but I find it really helpful as a visual for how these models work. So let's take a sort of contrived example. Let's say we've forgotten what the formula for BMI is, and we have a bunch of patients with a height and a weight, and we have their BMI, but we don't actually know the formula. Can we build a model that recovers the relationship from the data? This is a supervised task because we have labels, and it's a regression task because we're predicting a number.
So the model takes a first stab and draws an arbitrary line: y equals m x plus b, where m is the slope and b is the y-intercept. Those are the two things we need to tweak so that our orange line actually matches the blue dots, which are our real data. The red lines you see here are the residuals: the errors between the model's prediction and the actual data.
So the model takes a first stab and draws an arbitrary line: y equals m x plus b, where m is the slope and b is the y-intercept. Those are the two things we need to tweak so that our orange line actually matches the blue dots, which are our real data. The red lines you see here are the residuals: the errors between the model's prediction and the actual data.
So the model takes a first stab and draws an arbitrary line: y equals m x plus b, where m is the slope and b is the y-intercept. Those are the two things we need to tweak so that our orange line actually matches the blue dots, which are our real data. The red lines you see here are the residuals: the errors between the model's prediction and the actual data.
And the whole goal of learning, in machine learning, is to minimize that error. So we measure how far off we are, we nudge the slope and intercept a little, we measure again, and we repeat. And if you look on the right, you can see the loss goes down and down until it's essentially zero, and the orange line perfectly overlaps our data. And it lines up this perfectly because BMI really is just a formula of height and weight, so the model is literally rediscovering that formula from the data. Real outcomes are almost never this clean, but it's a nice way to watch the mechanism actually work. That loop (guess, measure error, adjust) is really the core of how basically every machine learning model learns.
And the whole goal of learning, in machine learning, is to minimize that error. So we measure how far off we are, we nudge the slope and intercept a little, we measure again, and we repeat. And if you look on the right, you can see the loss goes down and down until it's essentially zero, and the orange line perfectly overlaps our data. And it lines up this perfectly because BMI really is just a formula of height and weight, so the model is literally rediscovering that formula from the data. Real outcomes are almost never this clean, but it's a nice way to watch the mechanism actually work. That loop (guess, measure error, adjust) is really the core of how basically every machine learning model learns.
Obviously that last example was very clean; real-world data is never that perfect. So let's make up a new example. Same patient population, but now we're taking BMI as the input and we're trying to predict the severity of post-op complications on some arbitrary one-to-ten scale. If we try to draw a straight line through this, you can see it systematically underfits, and the worst deviation is out at the extremes, at the high BMIs and the low BMIs.
Obviously that last example was very clean; real-world data is never that perfect. So let's make up a new example. Same patient population, but now we're taking BMI as the input and we're trying to predict the severity of post-op complications on some arbitrary one-to-ten scale. If we try to draw a straight line through this, you can see it systematically underfits, and the worst deviation is out at the extremes, at the high BMIs and the low BMIs.
Obviously that last example was very clean; real-world data is never that perfect. So let's make up a new example. Same patient population, but now we're taking BMI as the input and we're trying to predict the severity of post-op complications on some arbitrary one-to-ten scale. If we try to draw a straight line through this, you can see it systematically underfits, and the worst deviation is out at the extremes, at the high BMIs and the low BMIs.
But what's interesting about machine learning is we're not stuck with one model. If linear regression isn't a great fit, we can swap in a polynomial regression, and you can keep tweaking parameters and trying different models to get the best fit for your data. Though (as we'll see later) best fit on your training data isn't always the goal.
But what's interesting about machine learning is we're not stuck with one model. If linear regression isn't a great fit, we can swap in a polynomial regression, and you can keep tweaking parameters and trying different models to get the best fit for your data. Though (as we'll see later) best fit on your training data isn't always the goal.
But what's interesting about machine learning is we're not stuck with one model. If linear regression isn't a great fit, we can swap in a polynomial regression, and you can keep tweaking parameters and trying different models to get the best fit for your data. Though (as we'll see later) best fit on your training data isn't always the goal.
But what's interesting about machine learning is we're not stuck with one model. If linear regression isn't a great fit, we can swap in a polynomial regression, and you can keep tweaking parameters and trying different models to get the best fit for your data. Though (as we'll see later) best fit on your training data isn't always the goal.
And it's not just polynomial regression. There's a whole other class of models we can toss in: trees, which really just define boundaries by splitting the data step by step. Support vector machines, which try to draw lines in a slightly different way. I'm not going to go deep into any of them. The point is really that for the same task, you can swap in a bunch of different model types: tweak them, play with them a little bit, and see which one works best for your data.
And it's not just polynomial regression. There's a whole other class of models we can toss in: trees, which really just define boundaries by splitting the data step by step. Support vector machines, which try to draw lines in a slightly different way. I'm not going to go deep into any of them. The point is really that for the same task, you can swap in a bunch of different model types: tweak them, play with them a little bit, and see which one works best for your data.
And it's not just polynomial regression. There's a whole other class of models we can toss in: trees, which really just define boundaries by splitting the data step by step. Support vector machines, which try to draw lines in a slightly different way. I'm not going to go deep into any of them. The point is really that for the same task, you can swap in a bunch of different model types: tweak them, play with them a little bit, and see which one works best for your data.
And it's not just polynomial regression. There's a whole other class of models we can toss in: trees, which really just define boundaries by splitting the data step by step. Support vector machines, which try to draw lines in a slightly different way. I'm not going to go deep into any of them. The point is really that for the same task, you can swap in a bunch of different model types: tweak them, play with them a little bit, and see which one works best for your data.
So why not just use linear regression for everything? Well, data can get complicated. Let's take another made-up example. Each dot here is a patient: the x-axis is their potassium, the y-axis is their magnesium, and the color shows whether they had an arrhythmia. You can imagine that if both values are in the normal range in the middle, the patient's unlikely to have an arrhythmia, but if the potassium goes too high or the magnesium goes too low, you start seeing arrhythmias.
So why not just use linear regression for everything? Well, data can get complicated. Let's take another made-up example. Each dot here is a patient: the x-axis is their potassium, the y-axis is their magnesium, and the color shows whether they had an arrhythmia. You can imagine that if both values are in the normal range in the middle, the patient's unlikely to have an arrhythmia, but if the potassium goes too high or the magnesium goes too low, you start seeing arrhythmias.
If you try to draw a straight line through this to classify arrhythmia versus no arrhythmia, no line is really going to work. So you'd need something more powerful (an SVM, a tree, a neural network), something that can see the data in a different way and draw non-linear boundaries.
If you try to draw a straight line through this to classify arrhythmia versus no arrhythmia, no line is really going to work. So you'd need something more powerful (an SVM, a tree, a neural network), something that can see the data in a different way and draw non-linear boundaries.
So, we've talked about a lot of things, but we haven't actually talked about neural networks yet, and I feel like that's what most people come into machine learning wanting to know about. At the end of the day, a neural network is just another algorithm, and it uses a lot of the same approaches we've already seen. It's really just multiple perceptrons stacked on top of each other. A perceptron is a very simple model: it takes multiple inputs, like a patient's height, weight, and age, assigns weights to each, sums them up, passes them through an activation function, and produces an output usually between zero and one. And just like our previous models, during training it tweaks those weights (W1, W2, W3) to figure out which inputs matter more.
So, we've talked about a lot of things, but we haven't actually talked about neural networks yet, and I feel like that's what most people come into machine learning wanting to know about. At the end of the day, a neural network is just another algorithm, and it uses a lot of the same approaches we've already seen. It's really just multiple perceptrons stacked on top of each other. A perceptron is a very simple model: it takes multiple inputs, like a patient's height, weight, and age, assigns weights to each, sums them up, passes them through an activation function, and produces an output usually between zero and one. And just like our previous models, during training it tweaks those weights (W1, W2, W3) to figure out which inputs matter more.
So, we've talked about a lot of things, but we haven't actually talked about neural networks yet, and I feel like that's what most people come into machine learning wanting to know about. At the end of the day, a neural network is just another algorithm, and it uses a lot of the same approaches we've already seen. It's really just multiple perceptrons stacked on top of each other. A perceptron is a very simple model: it takes multiple inputs, like a patient's height, weight, and age, assigns weights to each, sums them up, passes them through an activation function, and produces an output usually between zero and one. And just like our previous models, during training it tweaks those weights (W1, W2, W3) to figure out which inputs matter more.
A single perceptron is basically a straight line, useful, but limited. Stack them into layers and something new happens. Each hidden layer combines the one before it into a richer feature. The classic example is handwritten digits: you feed in the pixels of a scanned number, and the first hidden layer learns to spot edges, the next combines edges into strokes and loops, and the output layer has ten neurons (one for each digit, zero through nine), and the one that lights up is the model's answer. Nobody wrote a rule that says "this is a seven"; the network learned the features from examples. Reading a messy digit is effortless for your visual system and genuinely hard to hand-code, which is exactly the kind of problem machine learning is for.
A single perceptron is basically a straight line, useful, but limited. Stack them into layers and something new happens. Each hidden layer combines the one before it into a richer feature. The classic example is handwritten digits: you feed in the pixels of a scanned number, and the first hidden layer learns to spot edges, the next combines edges into strokes and loops, and the output layer has ten neurons (one for each digit, zero through nine), and the one that lights up is the model's answer. Nobody wrote a rule that says "this is a seven"; the network learned the features from examples. Reading a messy digit is effortless for your visual system and genuinely hard to hand-code, which is exactly the kind of problem machine learning is for.
A single perceptron is basically a straight line, useful, but limited. Stack them into layers and something new happens. Each hidden layer combines the one before it into a richer feature. The classic example is handwritten digits: you feed in the pixels of a scanned number, and the first hidden layer learns to spot edges, the next combines edges into strokes and loops, and the output layer has ten neurons (one for each digit, zero through nine), and the one that lights up is the model's answer. Nobody wrote a rule that says "this is a seven"; the network learned the features from examples. Reading a messy digit is effortless for your visual system and genuinely hard to hand-code, which is exactly the kind of problem machine learning is for.
A single perceptron is basically a straight line, useful, but limited. Stack them into layers and something new happens. Each hidden layer combines the one before it into a richer feature. The classic example is handwritten digits: you feed in the pixels of a scanned number, and the first hidden layer learns to spot edges, the next combines edges into strokes and loops, and the output layer has ten neurons (one for each digit, zero through nine), and the one that lights up is the model's answer. Nobody wrote a rule that says "this is a seven"; the network learned the features from examples. Reading a messy digit is effortless for your visual system and genuinely hard to hand-code, which is exactly the kind of problem machine learning is for.
I'm not going to go too deep into this; honestly it could take another twenty minutes. The video I'd recommend watching after this is 3Blue1Brown's "But What Is a Neural Network?" on YouTube. He walks through this exact example and visualizes it way better than I ever could.
I'm not going to go too deep into this; honestly it could take another twenty minutes. The video I'd recommend watching after this is 3Blue1Brown's "But What Is a Neural Network?" on YouTube. He walks through this exact example and visualizes it way better than I ever could.
I'm not going to go too deep into this; honestly it could take another twenty minutes. The video I'd recommend watching after this is 3Blue1Brown's "But What Is a Neural Network?" on YouTube. He walks through this exact example and visualizes it way better than I ever could.
I'm not going to go too deep into this; honestly it could take another twenty minutes. The video I'd recommend watching after this is 3Blue1Brown's "But What Is a Neural Network?" on YouTube. He walks through this exact example and visualizes it way better than I ever could.
If you look at the loss curve on the right, it starts off very high and goes down and down as the model learns. And if you watch the weights in the top right, you'll see them start off jumbled and then stabilize toward whatever value makes the loss the smallest. So it's really just the same loop again: guess, measure, adjust.
The reason neural networks are so cool is that when you stack multiple perceptrons on top of each other, you can draw really complex boundaries between different classes of data. So let's say we have this data (I can't think of a real clinical example that looks exactly like this, but bear with me) and we're trying to separate the red points from the blue points.
You'll see each perceptron in the hidden layer is drawing its own orange line, and when you combine all those lines you get this green boundary, which is really a combination of the orange ones. And once again, the loss starts off high and comes down as it learns, until it's classifying everything correctly. Same learning loop, dramatically more expressive result.
We've talked about how these models can be applied to different data types: tabular data, imaging, text, signals. We've talked about a few different model families. The one thing we haven't really gotten into is large language models, which is a bit out of scope for an intro. But the point is these models can be swapped in and out depending on your data and the kind of performance you want. And, you know, what we're usually looking for out of all of this is specific kinds of outputs: risk scores, diagnostic aids, warning systems, ambient scribes. Those are the actual tools you're seeing built out of these building blocks.
There's a concept that I feel like isn't talked about enough, at least among clinicians, but in machine learning it's very well known; it's called overfitting. The way I'd sort of think about it is, remember internal validity versus external validity from clinical studies? It's kind of the same idea. You can build a model that predicts every point in your training set perfectly (draws a line right through all of them) and then you deploy it in the real world and it only gets, you know, twenty percent of new patients right. That's overfitting.
There's a concept that I feel like isn't talked about enough, at least among clinicians, but in machine learning it's very well known; it's called overfitting. The way I'd sort of think about it is, remember internal validity versus external validity from clinical studies? It's kind of the same idea. You can build a model that predicts every point in your training set perfectly (draws a line right through all of them) and then you deploy it in the real world and it only gets, you know, twenty percent of new patients right. That's overfitting.
The classical way to fight this is to split your data three ways. There's the training data: that's what the model actually learns from. There's the validation data, which should never be part of the training data; you use it to tune things like hyperparameters. And then there's the test data, which you only touch once, at the very end, to get an honest sense of how the model will perform in the real world. If you accidentally train on data that shows up in your test set, your model will look way better than it actually is.
The classical way to fight this is to split your data three ways. There's the training data: that's what the model actually learns from. There's the validation data, which should never be part of the training data; you use it to tune things like hyperparameters. And then there's the test data, which you only touch once, at the very end, to get an honest sense of how the model will perform in the real world. If you accidentally train on data that shows up in your test set, your model will look way better than it actually is.
The classical way to fight this is to split your data three ways. There's the training data: that's what the model actually learns from. There's the validation data, which should never be part of the training data; you use it to tune things like hyperparameters. And then there's the test data, which you only touch once, at the very end, to get an honest sense of how the model will perform in the real world. If you accidentally train on data that shows up in your test set, your model will look way better than it actually is.
A few clinical things to think about. You probably don't want the same patient to show up in both your training and test data, even if the data points are different, because then the model is really just recognizing that patient. It might also make sense to split by time (train on earlier data and test on later data) because that's closer to how you'll actually deploy it. And if you can, the best validation is on data from a completely different health system.
A few clinical things to think about. You probably don't want the same patient to show up in both your training and test data, even if the data points are different, because then the model is really just recognizing that patient. It might also make sense to split by time (train on earlier data and test on later data) because that's closer to how you'll actually deploy it. And if you can, the best validation is on data from a completely different health system.
Okay, so for judging these models: the good news is you already know most of this. Sensitivity, specificity, PPV, NPV, the ROC curve, it's the same machinery we use to talk about any diagnostic test. Machine learning doesn't reinvent it; it mostly just renames it. So when you're reading a machine learning paper and you see terms like precision, recall, and F1 score, those might feel like new concepts, but they really aren't. Precision is just the same thing as PPV. Recall is actually just sensitivity. And F1 score is just a combined number: it takes the harmonic mean of precision and recall. So when you see these, don't freak out. It's the same stuff we already know, just with different words because it's a different discipline.
A quick warning about accuracy. Accuracy sounds great, but it can really mislead you when the outcome is rare. If only five percent of your patients have the complication, a "model" that just says "no complication" for everybody scores ninety-five percent. So accuracy on its own tells you almost nothing; you really want to look at sensitivity and PPV for the class you actually care about.
A quick warning about accuracy. Accuracy sounds great, but it can really mislead you when the outcome is rare. If only five percent of your patients have the complication, a "model" that just says "no complication" for everybody scores ninety-five percent. So accuracy on its own tells you almost nothing; you really want to look at sensitivity and PPV for the class you actually care about.
And then the other thing to keep in mind: PPV, or precision, is influenced by prevalence. The exact same model with the same sensitivity and specificity can give you a very different PPV in a screening clinic versus in an ICU. So when you're reading a number in a paper, always ask what population it came from.
One thing I don't think gets discussed enough: a prediction isn't a decision. The model just spits out a probability, say a sixty percent chance of the complication. Somebody still has to pick the cutoff where you actually act on it, and that choice is a real clinical lever. Set the threshold low and you catch almost every case, but you flood people with false alarms. Set it high and the false alarms drop, but you start missing real ones. Same model, same patients: very different behavior depending on where you draw the line. And it's really the clinical system around the model, not the model itself, that decides what to do with that number.
And there are two questions here that matter a lot. Should the model act on its own, or should it just flag something for a clinician to review? And how does it fit into the existing workflow: does it show up in the EMR at the right moment, or is it just another inbox notification nobody reads? These implementation details barely get mentioned in papers, but they matter as much as accuracy for whether the model is actually useful.
And there are two questions here that matter a lot. Should the model act on its own, or should it just flag something for a clinician to review? And how does it fit into the existing workflow: does it show up in the EMR at the right moment, or is it just another inbox notification nobody reads? These implementation details barely get mentioned in papers, but they matter as much as accuracy for whether the model is actually useful.
So I have three quick stories I want to walk through: real cases where the model didn't do the job it was told to.
So I have three quick stories I want to walk through: real cases where the model didn't do the job it was told to.
First story: this is a real one, Obermeyer and colleagues, Science, twenty-nineteen. A widely used algorithm, the kind that decides who gets enrolled in extra care-management programs, touching millions of patients, was trained to predict a patient's future healthcare costs, as a stand-in for how sick they are. Reasonable-sounding proxy. But at the same predicted cost, Black patients were considerably sicker than White patients. The reason: the same illness generates less recorded cost for patients who, through access barriers, use less care. The model learned costs faithfully; costs just weren't health. The authors estimated that fixing it would raise the share of Black patients flagged for extra help from about eighteen percent to forty-seven. The model wasn't broken. The label was.
First story: this is a real one, Obermeyer and colleagues, Science, twenty-nineteen. A widely used algorithm, the kind that decides who gets enrolled in extra care-management programs, touching millions of patients, was trained to predict a patient's future healthcare costs, as a stand-in for how sick they are. Reasonable-sounding proxy. But at the same predicted cost, Black patients were considerably sicker than White patients. The reason: the same illness generates less recorded cost for patients who, through access barriers, use less care. The model learned costs faithfully; costs just weren't health. The authors estimated that fixing it would raise the share of Black patients flagged for extra help from about eighteen percent to forty-seven. The model wasn't broken. The label was.
Story two: Zech and colleagues, twenty-eighteen. They trained a model to detect pneumonia on chest x-rays. On held-out images from its own hospitals it did well, an AUC around point-nine-three. Deploy it at a new hospital and it fell to point-eight-two. Here's the interesting part. The model could tell which hospital an x-ray came from with about ninety-nine-point-nine percent accuracy, off cues like the "portable" marker stamped on films shot at the bedside, which get ordered on sicker patients. And pneumonia was thirty-four percent of scans at one hospital versus one percent at another. So "which hospital" alone predicts pneumonia. The model learned that shortcut, not the disease: right answer, wrong reason, and the reason didn't travel. The general name for this is shortcut learning: the model finds the easiest pattern that explains the training data, not the one you were hoping it would find.
Story two: Zech and colleagues, twenty-eighteen. They trained a model to detect pneumonia on chest x-rays. On held-out images from its own hospitals it did well, an AUC around point-nine-three. Deploy it at a new hospital and it fell to point-eight-two. Here's the interesting part. The model could tell which hospital an x-ray came from with about ninety-nine-point-nine percent accuracy, off cues like the "portable" marker stamped on films shot at the bedside, which get ordered on sicker patients. And pneumonia was thirty-four percent of scans at one hospital versus one percent at another. So "which hospital" alone predicts pneumonia. The model learned that shortcut, not the disease: right answer, wrong reason, and the reason didn't travel. The general name for this is shortcut learning: the model finds the easiest pattern that explains the training data, not the one you were hoping it would find.
Story two: Zech and colleagues, twenty-eighteen. They trained a model to detect pneumonia on chest x-rays. On held-out images from its own hospitals it did well, an AUC around point-nine-three. Deploy it at a new hospital and it fell to point-eight-two. Here's the interesting part. The model could tell which hospital an x-ray came from with about ninety-nine-point-nine percent accuracy, off cues like the "portable" marker stamped on films shot at the bedside, which get ordered on sicker patients. And pneumonia was thirty-four percent of scans at one hospital versus one percent at another. So "which hospital" alone predicts pneumonia. The model learned that shortcut, not the disease: right answer, wrong reason, and the reason didn't travel. The general name for this is shortcut learning: the model finds the easiest pattern that explains the training data, not the one you were hoping it would find.
And this ties together everything we've been talking about: a model can drop the moment you move it somewhere new, and often nothing tells you it happened. So before you trust one, you want to know where the data came from, how it was validated, whether it actually matches your setting, and how you'll notice when it starts to slip. That's exactly the checklist we'll end on.
And this ties together everything we've been talking about: a model can drop the moment you move it somewhere new, and often nothing tells you it happened. So before you trust one, you want to know where the data came from, how it was validated, whether it actually matches your setting, and how you'll notice when it starts to slip. That's exactly the checklist we'll end on.
Alright, to quickly summarize: a few main takeaways. One, just because a model performs well in a paper or on some training data or a validation set, doesn't mean it's going to keep performing well on your patient population. You have to check. Two, the right metric depends on the problem; accuracy alone will mislead you. Three, a model can absolutely be right for the wrong reason.
Four, before trusting a model, ask what data it was trained on, how it was validated, and whether that matches your clinical setting. And five (and this is really the one I think matters most) a model might be perfect on paper, but how it's deployed, the harness, how it hooks into the EMR and the clinician workflow, that's what actually drives whether it works in the real world.
Four, before trusting a model, ask what data it was trained on, how it was validated, and whether that matches your clinical setting. And five (and this is really the one I think matters most) a model might be perfect on paper, but how it's deployed, the harness, how it hooks into the EMR and the clinician workflow, that's what actually drives whether it works in the real world.
So that's really the end of it. This was the first time I've built a lecture like this, and it's obviously still a work in progress. In Part Two we go deeper on large language models specifically: the chatbots and the ambient scribes you're actually running into. What they really are under the hood, why they confidently make things up, some of the misconceptions people have about them, what gets built around them to make them useful, and a rubric every clinician can use when evaluating these tools. So, I'd really appreciate any feedback, and thanks for watching.