Browse videos
Let's build GPT: from scratch, in code
Study materials
Materials for this selected video are in preparation. Watch the video and use the public learning guide while they are being prepared.
Slides are a PPT outline, not a finished slide deck.
Overview
Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.
Why HN discusses it
It is the go-to recommendation for learning how transformers and LLMs actually work, praised for its clarity and Karpathy's teaching.
Before you watch
Familiarity with Python, tensors, PyTorch and earlier makemore lessons is recommended. Choose this session when you can already train and inspect a small neural network.
A way to study this video
Implement one component at a time and check tensor shapes before continuing. Explain the role of attention using your own code, then connect the training objective back to the character-level model.
Follow the Andrej Karpathy learning guide · Original Zero to Hero course
Newest first
In discussion: SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch
The one you linked to is based on Karpathy's tutorial: [VIDEO] , except changing to train on tinystories in stead of Shakespeare. smolGPT also looks inspired by nanoGPT, also from Karpathy: [LINK]
In discussion: A ChatGPT clone, in 3000 bytes of C, backed by GPT-2 (2023)
Attention is all you need and GPT-2 were well known at that point. Many might doubt whether this approach leads to "general" intelligence – depends on the definition. BTW, Karpathy has a nice video tutorial about building an LLM: [VIDEO]
In discussion: Building LLMs from the Ground Up: A 3-Hour Coding Workshop
Excuse my ignorance, is this different from Andrej Karpathy [VIDEO] Anyway I will watch it tonight before bed. Thank you for sharing.
In discussion: AI's $600B Question
That's... my point. You're agreeing with me. Siri and Alexa are bad . Very, very bad. They pretend to understand spoken English, but they don't , because they're just a huge set of hard-coded rules written out one-at-a-time by enormous numbers of very expensive developers. This is the 1990s approach to AI: Fuzzy logic, heuristics, solvers, dynamic programming, etc... That approach has been thoroughly blown out of the…That's... my point. You're agreeing with me. Siri and Alexa are bad . Very, very bad. They pretend to understand spoken English, but they don't , because they're just a huge set of hard-coded rules written out one-at-a-time by enormous numbers of very expensive developers. This is the 1990s approach to AI: Fuzzy logic, heuristics, solvers, dynamic programming, etc... That approach has been thoroughly blown out of the water by Transformers, which does all of that and much more with a thousand lines of code that can be banged out in a couple of hours by one guy while talking in a YouTube video: [VIDEO] Transformers will revolutionise this entire space, and more that don't even exist yet as a market. Take for example humanoid robots: Boston Dynamics has had the hardware working well enough for a decade, but not the software . You can't walk up to one of their robots, point at something, and tell the robot to complete a task. It can't understand what it is seeing, and can't understand English instructions. Programming that in with traditional AI methods would take man-millenia of effort, and might never work well enough . If we could speed up GPT-4o (the one with vision) to just 10x or 50x its current speed, with some light fine-tuning it could control a humanoid robot right now with a level of understanding comparable to C3P0 from Star Wars!
In discussion: Implementing a ChatGPT-like LLM from scratch, step by step
+1, Andrey is an amazing educator! I'd also recommend his [VIDEO] and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high.
Showing 5 of 20 comments