学习资料
浏览视频
返回视频列表
从零构建 GPT：用代码理解模型
学习资料
这个精选视频的资料正在准备中，可以先观看视频并阅读学习导读。
PPT 资料为文字大纲，尚非完整演示文稿。
视频概览
Andrej Karpathy 用约两小时，以 PyTorch 从零实现 GPT，参考 Attention Is All You Need 和 GPT-2/3，并说明它与 ChatGPT 的关系。课程默认观众熟悉前面的 makemore 视频。
HN 讨论的原因
HN 常推荐它帮助理解 Transformer 与大语言模型的实际工作原理，尤其赞赏讲解的清晰度。
观看前准备
建议先熟悉 Python、张量、PyTorch 和前面的 makemore 课程。在已经能训练并检查小型神经网络时，再开始这一课。
学习建议
逐个实现组件，每一步都先检查张量形状。用自己的代码解释注意力的作用，再把训练目标与字符级模型联系起来。
最新优先
所在讨论： SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch
The one you linked to is based on Karpathy's tutorial: [VIDEO] , except changing to train on tinystories in stead of Shakespeare. smolGPT also looks inspired by nanoGPT, also from Karpathy: [LINK]
所在讨论： A ChatGPT clone, in 3000 bytes of C, backed by GPT-2 (2023)
Attention is all you need and GPT-2 were well known at that point. Many might doubt whether this approach leads to "general" intelligence – depends on the definition. BTW, Karpathy has a nice video tutorial about building an LLM: [VIDEO]
所在讨论： Building LLMs from the Ground Up: A 3-Hour Coding Workshop
Excuse my ignorance, is this different from Andrej Karpathy [VIDEO] Anyway I will watch it tonight before bed. Thank you for sharing.
所在讨论： AI's $600B Question
That's... my point. You're agreeing with me. Siri and Alexa are bad . Very, very bad. They pretend to understand spoken English, but they don't , because they're just a huge set of hard-coded rules written out one-at-a-time by enormous numbers of very expensive developers. This is the 1990s approach to AI: Fuzzy logic, heuristics, solvers, dynamic programming, etc... That approach has been thoroughly blown out of the…That's... my point. You're agreeing with me. Siri and Alexa are bad . Very, very bad. They pretend to understand spoken English, but they don't , because they're just a huge set of hard-coded rules written out one-at-a-time by enormous numbers of very expensive developers. This is the 1990s approach to AI: Fuzzy logic, heuristics, solvers, dynamic programming, etc... That approach has been thoroughly blown out of the water by Transformers, which does all of that and much more with a thousand lines of code that can be banged out in a couple of hours by one guy while talking in a YouTube video: [VIDEO] Transformers will revolutionise this entire space, and more that don't even exist yet as a market. Take for example humanoid robots: Boston Dynamics has had the hardware working well enough for a decade, but not the software . You can't walk up to one of their robots, point at something, and tell the robot to complete a task. It can't understand what it is seeing, and can't understand English instructions. Programming that in with traditional AI methods would take man-millenia of effort, and might never work well enough . If we could speed up GPT-4o (the one with vision) to just 10x or 50x its current speed, with some light fine-tuning it could control a humanoid robot right now with a level of understanding comparable to C3P0 from Star Wars!
所在讨论： Implementing a ChatGPT-like LLM from scratch, step by step
+1, Andrey is an amazing educator! I'd also recommend his [VIDEO] and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high.
已显示 5 / 20 条评论