Browse videos
Deep Dive into LLMs like ChatGPT
Study materials
Materials for this selected video are in preparation. Watch the video and use the public learning guide while they are being prepared.
Slides are a PPT outline, not a finished slide deck.
Overview
Andrej Karpathy's three-hour-plus general-audience deep dive into how ChatGPT-style LLMs are built, covering the full training stack from pretraining to post-training. It also gives mental models for their quirks, like hallucination, and how to use them well.
Why HN discusses it
HN's most frequent recommendation for a clear, approachable foundation in how LLMs work without heavy math.
Before you watch
No model-training experience is required for a conceptual first pass. This three-and-a-half-hour video suits learners who want a broad account of LLMs before writing code.
A way to study this video
Split the video into several sessions. After each section, distinguish training from inference and write one question about what the model can and cannot learn. Use the shorter introduction if you need an orientation first.
Follow the Andrej Karpathy learning guide · Original Zero to Hero course
Newest first
In discussion: Show HN: How LLMs Work – Interactive visual guide based on Karpathy's lecture
I agreed when I read your comment, but that turned out to be almost directly from the video [VIDEO] From around the 2:20 mark he says: “[…] actually ends up being only about 44 TB of disk space. You can get a USB stick for like a TB very easily, or I think this could fit on a single hard drive almost today” So it’s just slightly altered from what was said in the original video. And the LLM rewritten version of it…I agreed when I read your comment, but that turned out to be almost directly from the video [VIDEO] From around the 2:20 mark he says: “[…] actually ends up being only about 44 TB of disk space. You can get a USB stick for like a TB very easily, or I think this could fit on a single hard drive almost today” So it’s just slightly altered from what was said in the original video. And the LLM rewritten version of it also says “roughly” where he said “almost”, and I guess 44 TB is pretty roughly or pretty almost 32 TB. Although I’d still personally probably put it as “can fit on a pair of decently sized hard drives today” (for example across two 24 TB drives). Regardless, it’s close enough to what was said in the source video that it’s not something the LLM just made up out of nowhere.
In discussion: Microgpt
Even if you have some basic understanding of how LLMs work, I highly recommend Karpathy’s intro to LLMs videos on YouTube. - [VIDEO] - [LINK]
In discussion: Ask HN: What would you recommend a vibe coder learn about how all this works?
I highly recommend Andrej Karpathy's Deep Dive into LLMs like ChatGPT: [VIDEO] . It's a 3.5 hour general intro to the topic that I think does a great job of making sense of the tech and how it works under the hood, without needing to build up a foundation of math and programming.
In discussion: Neural Networks: Zero to Hero
This is a good resource, however for about 99.99% of people, you are most likely to just use a foundation model like ChatGPT, Claude, Gemini etc. so this knowledge/training will get you neither here or there. I would suggest you look into another Karpathy's video -- Deep Dive into LLMs like ChatGPT. [VIDEO]
In discussion: Using secondary school maths to demystify AI
That's post-training No, it is not. Read the paper. They are discussing an emergent property of the context itself: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model." I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th…That's post-training No, it is not. Read the paper. They are discussing an emergent property of the context itself: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model." I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator. So am I. See, for example, Karpathy's discussion of native computation: [VIDEO] (starts at 1:46:56). The first few tokens in the context actually serve as some sort of substrate for general computation. I don't pretend to understand that, and it may still be something of an open research topic, but it's one more unexpected emergent property of transformers. You'd be crazy to trust that property at this stage -- at the time Karpathy was making the video, he needed to explicitly tell the model to "Use code" if he didn't want it to just make up solutions to more complex problems -- but you'd also be crazy to trust answers from a 5th-grader who just learned long division last week. Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly. Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.
Showing 5 of 18 comments