学习资料
浏览视频
返回视频列表
手写 makemore：从小实验进入语言建模
学习资料
这个精选视频的资料正在准备中，可以先观看视频并阅读学习导读。
PPT 资料为文字大纲，尚非完整演示文稿。
视频概览
Andrej Karpathy 从零构建二元字符级语言模型，介绍 torch.Tensor，以及训练、采样和损失评估的基本流程。这是 makemore 系列迈向 GPT 类 Transformer 的第一步。
HN 讨论的原因
HN 推荐它作为从技术层面理解语言模型的方法，认为虽然较长，但值得投入时间。
观看前准备
熟悉 Python 和 micrograd 中的概念会有帮助。这一课适合准备从标量神经网络示例迈向字符级语言建模的学习者。
学习建议
复现小型姓名生成实验，改变一个输入并预测其影响。尝试实现 GPT 前，先完成 Zero to Hero 课程的中间章节。
最新优先
所在讨论： The n-gram Language Model
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [LINK] It appears to be a generalized version of the bi-gram project, which he links to here: * [VIDEO] It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.
所在讨论： Show HN: Next-token prediction in JavaScript
You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [LINK] - Highly recommend this one [VIDEO] - [LINK] - [LINK] 1: ex: [LINK]
所在讨论： GPT in 500 Lines of SQL
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] [VIDEO]
所在讨论： Markov Chains are the Original Language Models
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. [VIDEO]