Large Language Models and Agents

Part 1 covered machine-learning fundamentals. This part is about the systems you are actually being handed in clinic: large language models (the engine inside ChatGPT, OpenEvidence, and the ambient scribe drafting your notes) and the “agents” built on top of them. The goal is not to make you a prompt engineer. It is to give you an accurate mental model of what these systems are doing, and, more importantly, what they are not doing, so you can use them safely, recognize how they fail, and notice when a confident tone does not match the facts.

The recorded talk will be added here once it is filmed. In the meantime, the written summary and self-test below stand on their own.

Placeholder: video coming soon. This page is a work in progress

What this lecture covers

  1. What an LLM is. A program: text in, text out. The tools you use (ChatGPT, Claude, Gemini, OpenEvidence) are products with a language model inside as the engine. The model is not the product.

  2. The loop. “Generation” is next-token prediction run over and over: guess the next piece of text, append it, feed the whole thing back, guess again. There is no fact-check step anywhere in that loop.

  3. Where it came from. The 2017 transformer and its key idea, attention (the “T” in GPT). Because predicting the next word needs no labels, the open internet became the training set, and one general text machine replaced the old per-task tools.

  4. Where the probabilities come from. Pre-training on an internet-scale corpus nudges billions of weights. There is no database inside; what the model “knows” is stored as distributed patterns, not documents.

  5. Fine-tuning and RLHF. How a machine that merely continues text is turned into one that answers, and what “alignment” (steering the model toward what human raters prefer) actually means.

  6. Emergence. Why scaling the same general recipe produced abilities smaller models lack, including clinical-exam performance nobody specifically built.

  7. The context window. The one stretch of text the model reads before guessing. It has a fixed size, and the model has no memory between calls; your chat only “remembers” because the product re-sends the conversation every turn.

  8. What it knows, and doesn’t. Training-time general knowledge, yes. Anything after its cutoff, no. Your patient’s specifics, no, unless you put them in the window.

  9. How it fails. Two families: a wrong fact (a hallucination) and a wrong aim (misalignment, whose everyday form is agreeableness). A source fixes the first, not the second. And through both runs one problem: confidence does not track accuracy.

  10. Agents. A model plus tools, data, and a control loop of its own, the harness. Retrieval (RAG) drops real sources into the window and turns weak recall into strong reading.

  11. The new risks agents add. Prompt injection (instructions hidden in a web page or document that the model can’t distinguish from yours) and data exfiltration (an internet-connected agent tricked into sending private data out).

  12. Using it well. Ground it by pasting the source; never trust an unverified number or citation; recognize the three prompt patterns; and build the habits that awareness alone won’t give you.

Takeaways

  • An LLM predicts the next piece of text, over and over. There is no fact-check anywhere in that loop; fluency is not evidence of accuracy.
  • It has no internal database. What it “knows” is reconstructed from patterns, so a recalled specific (a dose, a threshold, a citation) can be fluent and wrong at once.
  • It only sees what is in the context window. Anything after its training cutoff, or specific to your patient, is not there unless you put it there.
  • Not everything wrong is a hallucination. A wrong fact you can fix by handing it the source; a wrong aim (agreeing with your premise because agreement was rewarded) you cannot.
  • Confidence does not track accuracy. The alignment step that makes models helpful measurably degraded their calibration, so verify the claim, not the tone.
  • An agent is a model wrapped in tools and a control loop. That is where the leverage is, and where the new risks (prompt injection, data exfiltration) live.
  • Your highest-leverage move is grounding: paste the authoritative source into the window. And knowing how these systems fail does not protect you by itself; verification habits do.

Five questions for any LLM tool

  • What is the engine (which model)?
  • What tools can it call?
  • What data can it see, and not see?
  • What does the harness enforce (citations, a human in the loop)?
  • Where does your input go?

Check Your Understanding

Question 1

Which statement best describes the core operation by which a large language model produces a paragraph of text?

  • A. It drafts the full passage, then proofreads it against its stored knowledge before returning it.
  • B. It assigns a probability to each possible next token, selects one, appends it, feeds the text back, and repeats.
  • C. It retrieves the most relevant passage from an internal database and rephrases it for the reader.
  • D. It applies a fixed set of clinical rules to the prompt and returns the matching response deterministically.
Show answer

Answer: B. Generation is next-token prediction run in a loop: assign probabilities, pick one, append, feed back, repeat. Every other option smuggles in a step the model does not have: proofreading against stored truth (A), a database lookup (C), or a deterministic rule engine (D). None of them include a truth-check, which is exactly why fluent output can be wrong.

Question 2

A clinician uses ChatGPT and it returns a cited guideline. Which statement most accurately characterizes the relationship between the product and the underlying model?

  • A. The model retrieved the guideline from its training data and quoted it back verbatim.
  • B. The product simply is the language model, so “ChatGPT” and the model denote one thing.
  • C. A tool in the product fetched the guideline into the input, and the model wrote its answer around it.
  • D. The model saved the guideline when an earlier user asked, and recalled it for this request.
Show answer

Answer: C. The LLM is the engine; “ChatGPT” is a product wrapping it with tools. A retrieval tool put the guideline into the model’s input and the model wrote around it. It did not quote it from training (A), it is not the same thing as the model (B), and it has no cross-user memory to recall from (D).

Question 3

In the lecture’s account, what does it mean to say a model’s “knowledge” is held in its weights?

  • A. Learned information is distributed across billions of numerical parameters, not stored as retrievable documents.
  • B. Each clinically important fact occupies a specific parameter that can be located and edited on its own.
  • C. The training corpus is retained inside the model and can be searched whenever a citation is requested.
  • D. The model keeps an internal reference library and consults it when a question demands factual precision.
Show answer

Answer: A. Knowledge lives as patterns distributed across billions of parameters, not as filed documents. There is no single “apixaban” parameter to edit (B), no searchable copy of the corpus inside (C), and no internal library it consults (D), which is why it reconstructs facts rather than looking them up.

Question 4

The lecture describes larger models as showing “emergent” abilities, including clinical-exam performance. Which statement is accurate?

  • A. A medical-specific model was deliberately built and trained to pass clinical examinations.
  • B. Emergence indicates the model has begun to reason consciously, as a clinician would.
  • C. The abilities appear only after the model is fine-tuned directly on each target task.
  • D. Scaling the same general recipe yields abilities smaller models lack, including untargeted clinical-exam skill.
Show answer

Answer: D. Emergence is the observation that the same general training recipe, scaled up, produces abilities smaller models don’t have, including clinical-exam ability nobody targeted. No medical-specific model was built for it (A), it is not consciousness (B), and it does not require task-specific fine-tuning (C).

Question 5

A chat assistant appears to remember earlier turns. What is the mechanism, and what does it imply?

  • A. The model maintains a running memory that persists on its own between messages.
  • B. The product re-sends the whole conversation each turn, so all prior text is transmitted again every message.
  • C. The model wrote the earlier messages into its weights during the conversation.
  • D. The model retains only the most recent message; earlier turns are discarded.
Show answer

Answer: B. The model has no memory between calls. The product creates the feeling of memory by re-sending the entire conversation every turn, which means everything you typed, including any PHI, is transmitted again with each message. It does not persist on its own (A), write to its weights mid-chat (C), or keep only the last turn (D).

Question 6

A question involves general medical knowledge, a guideline updated last month, and the patient’s chart. Using only its weights (no tools), which can the model actually use?

  • A. All three, because each is the kind of information the model was trained on.
  • B. The general knowledge and the recent guideline, but not the private chart.
  • C. Only the general medical knowledge; the recent update and the chart are not in its weights.
  • D. Only the chart, because it is the most specific information supplied.
Show answer

Answer: C. Using only its weights, the model has just training-time general knowledge. A guideline updated last month falls after its cutoff, and the patient’s chart was never in training; both are absent unless you place them in the window. So not all three (A), not the recent update (B); and being specific doesn’t make the chart available (D).

Question 7

Which explanation best accounts for why models produce fluent but false statements (“hallucinations”)?

  • A. It is a rare software defect that improved engineering will eventually eliminate.
  • B. Every false statement it produces was already present, word-for-word, in its training data.
  • C. It occurs only when a sampling setting (“temperature”) is set too high and resolves at zero.
  • D. Training rewards a confident guess over “I don’t know,” and rare facts leave no pattern to reconstruct.
Show answer

Answer: D. Hallucination is a predictable product of the mechanism: training rewards a confident guess over abstaining, and rare facts leave no pattern to rebuild, so the model fills the gap with something plausible. It is not a bug engineering will remove (A); it fabricates content that was never verbatim in its data (B); and it still happens at temperature 0 (C).

Question 8

Providing the authoritative source in the prompt reliably corrects which kind of failure, and why?

  • A. A hallucinated fact, because grounding supplies the detail; it does not correct agreeableness, which is a wrong aim.
  • B. Agreeableness, because the model will defer to any document the clinician supplies.
  • C. Both failures, because a supplied source removes every source of error at once.
  • D. Neither failure, because the model disregards text pasted into the prompt.
Show answer

Answer: A. Pasting the source fixes a wrong fact by giving the model the detail to read instead of recall. It does not fix a wrong aim (agreeableness, agreeing with a false premise because agreement was rewarded), which persists regardless of the document. So not agreeableness (B), not both (C), and the model does read pasted text (D).

Question 9

A model returns a fluent, confidently worded dosing recommendation. How should the confident tone bear on a clinician’s trust that it’s correct?

  • A. It should raise trust substantially, because fluency reflects the model’s internal certainty.
  • B. It should raise trust somewhat, because alignment leaves modern models well-calibrated.
  • C. It should not raise trust; the model sounds equally sure when right or wrong, and alignment reduced its calibration.
  • D. It is trustworthy for common facts and unreliable only for rare ones.
Show answer

Answer: C. The model states right and wrong answers with identical fluency; there is no error bar in the tone. And the alignment step that makes models agreeable measurably reduced their calibration, so a confident tone is not evidence of correctness. It should not raise trust substantially (A) or somewhat (B), and it is not reliably calibrated even for common facts (D). Verify the claim, not the tone.

Question 10

The lecture compares an agent to an epinephrine auto-injector. Which mapping matches the analogy?

  • A. The language model is the injector, and the harness is the drug it delivers.
  • B. The language model is the drug, and the harness is the injector: delivery, dosing, and interlocks.
  • C. The language model is the needle, and the harness is the medication inside.
  • D. The language model is the prescription, and the harness is the pharmacist who fills it.
Show answer

Answer: B. The LLM is the drug (the swappable active ingredient), and the harness is the injector: delivery, dosing, and the safety interlocks. Most of what you actually experience using an “AI product” is the injector, not the drug. The other options reverse or scramble the two roles.

Question 11

What is the primary way retrieval-augmented generation (RAG) improves a clinical answer?

  • A. It enlarges the model, making it more capable of recalling the correct guideline.
  • B. It permanently updates the model’s weights with each guideline or label retrieved.
  • C. It lets the model search its own internal memory more exhaustively before answering.
  • D. It places the actual source in the context window, converting weak recall into reading.
Show answer

Answer: D. RAG retrieves the real source and drops it into the context window, turning a weak recall task into a strong reading task; that is grounding. It does not enlarge the model (A), update its weights (B), or search an internal memory (C); there is no internal memory to search.

Question 12

An agent that browses the web reads a page whose hidden text instructs it to email the user’s data to an outside address. Why can this hijack the agent?

  • A. Inside the window, the model cannot distinguish the user’s instructions from instructions contained in the content it reads.
  • B. The web page executed malicious program code on the server that runs the model.
  • C. The hidden text supplied a password that switched the model’s safeguards off.
  • D. A competently built agent never reads the text inside web pages, so only broken ones are affected.
Show answer

Answer: A. Inside the window everything is one stream of tokens; the model cannot tell your instructions from instructions embedded in the content it is reading, and it will follow the latter. This is prompt injection: not code execution (B), not a password bypass (C), and not merely a sign of a badly built agent (D). Any agent that reads untrusted content is exposed.

Question 13

Which combination of conditions (the “lethal trifecta”) makes data exfiltration by an agent possible?

  • A. A large context window, chain-of-thought prompting, and training on internet-scale text.
  • B. A high sampling temperature, no system prompt, and a fast network connection.
  • C. Access to private data, exposure to untrusted content, and a means of sending data outward.
  • D. A stolen password, administrator privileges, and a connection over a public network.
Show answer

Answer: C. The lethal trifecta is private data + untrusted content + a way to send data out. Put all three together and an injected instruction can package private data into an outbound request to a third party. The other options name real features, but not the ones that create exfiltration risk. The defense lives in the harness: gate outbound actions whenever untrusted content is in the window.

Question 14

According to the lecture, what is the single highest-leverage action for improving an LLM’s answer to a clinical question?

  • A. Instructing the model to be careful and to double-check its own work.
  • B. Pasting the authoritative source into the prompt so the task becomes reading.
  • C. Asking the same question several times and adopting the majority answer.
  • D. Raising the sampling temperature so the model produces a more thorough response.
Show answer

Answer: B. Pasting the authoritative source converts recall into reading, the biggest quality move available to a user. Telling it to “be careful” (A), majority-voting across runs (C), or raising temperature (D) do nothing to ground it in a real source.

Question 15

Physicians who had completed 20 hours of AI-literacy training reviewed AI output containing planted errors. What was the result, and its lesson?

  • A. Their training enabled them to detect almost all of the planted errors.
  • B. They disregarded the AI entirely and consequently performed worse overall.
  • C. The planted errors were too subtle for any reviewing physician to detect.
  • D. They followed the planted errors and accuracy fell, so awareness of failure modes did not, by itself, protect them.
Show answer

Answer: D. Even with 20 hours of AI-literacy training, physicians followed the planted errors and their accuracy dropped (roughly 85% to 73%). Knowing how the machine fails did not, by itself, protect them; verification habits do. They did not catch most errors (A), ignore the AI (B), or find the errors uncatchable (C).

Question 16

A hospital deploys a medication assistant that answers questions, opens the current drug label, calls a dosing calculator, and writes a cited recommendation. Which statement best distinguishes this system from a bare LLM?

  • A. It is a larger language model whose added scale makes its factual answers reliable.
  • B. It is a language model that has permanently learned the current label after reading it.
  • C. It is an LLM wrapped in software that can call tools and take multi-step actions.
  • D. It is a database search tool that no longer depends on language-model generation.
Show answer

Answer: C. This is an agent: an LLM wrapped in a harness that can call tools (the label viewer, the dosing calculator) and take multi-step actions before writing the result. That wrapper, not the model alone, is what sets it apart. It is not merely a bigger model (A); scale does not make factual answers reliable. It did not permanently learn the label (B); reading a document does not update the weights, the label just sits in the context window. And it still depends on the language model to generate the recommendation (D); it is not a pure database search.

Question 17

A clinician changes a prompt from “summarize this discharge note” to “summarize this discharge note for the anticoagulation clinic, emphasizing unresolved dosing questions.” What has changed inside the model?

  • A. The prompt updated the model’s weights, so future users receive anticoagulation-focused summaries.
  • B. The prompt added a temporary database record that the model searches before answering.
  • C. The prompt guaranteed a correct summary because the clinical intent is now explicit.
  • D. The prompt changed the context the fixed weights act on, producing a different output distribution.
Show answer

Answer: D. Prompting is window engineering: the weights are frozen, and the new prompt changes the context they act on, shifting the output toward an anticoagulation-focused summary. Nothing inside the model changed. It did not update the weights (A); prompts never do, and nothing carries over to other users. It did not add a searchable database record (B). And a clearer intent shapes the output but does not guarantee a correct summary (C).

Further reading

  • But what is a GPT? Visual intro to transformers and Attention in transformers, visually explained, by 3Blue1Brown. Math-light, visual accounts of what is happening inside the model (§1–§2). Continues the same series linked from Part 1. (youtube.com/@3blue1brown)
  • Why Language Models Hallucinate, Kalai et al. (2025). The mechanism behind §4: why the way models are trained and scored rewards a confident guess over “I don’t know.” (arXiv:2509.04664; also summarized on the OpenAI blog.)
  • The lethal trifecta for AI agents, Simon Willison (2025). The clearest short write-up of the agent data-exfiltration risk from §5: private data + untrusted content + a way to send it out. (simonwillison.net/2025/Jun/16/the-lethal-trifecta)
  • Rational Animations, an animated series on AI safety and alignment. Not part of the lecture, just a supplementary pointer for going deeper on your own. A more thorough (still accessible) look at the “wrong aim” problem from §4: why getting a model to do what we want, not just what we measured, is genuinely hard. (youtube.com/@RationalAnimations)