AI 入门：从函数到 Transformer Intro to AI: From Functions to Transformers

1.1 早期思路：符号主义1.1 The early approach: Symbolic AI

把现实世界抽象成符号，再定好运算规则，函数算出的结果反过来解释世界。输入直角三角形的两条直角边，勾股定理告诉你斜边；输入质量和加速度，牛顿第二定律告诉你力。这就是人工智能最早的思路——符号主义（Symbolism）：相信万物背后都有一条能被人类明确写出来的规律。 Abstract the world into symbols, define the rules, and let the function's output explain reality. Feed in the two legs of a right triangle and the Pythagorean theorem gives you the hypotenuse; feed in mass and acceleration and Newton's second law gives you the force. That was AI's earliest approach — Symbolic AI: the belief that behind everything lies a rule humans can write down explicitly.

但这条路走到头了。识别一张图片是不是猫，对人类简单到爆炸，可要写成一段明确的程序规则，一下子就变成史诗级难题——猫有几万种姿势、光线、角度，规则根本列不完。连有明确语法和词典的翻译尚且做不到丝滑，更别说复杂多变的人类智能了。 But that road hit a dead end. Telling whether a photo shows a cat is trivially easy for a human — yet writing it as explicit program rules is an epic-level problem: cats come in endless poses, lighting, and angles, and the rules never end. Even translation, which has grammar books and dictionaries, never got smooth this way — let alone the full messiness of human intelligence.

1.2 换个思路：猜，然后不断逼近1.2 A new idea: guess, then keep getting closer

既然不知道函数长什么样，那就别硬找了。假设我们手里有一些 x 和 y 的数据点，想找出它们的函数关系。人类有史以来最具智慧的办法登场了：猜。先随便猜一条直线 ŷ = w·x + b，然后一点点调整 w（斜率）和 b（截距），让直线越来越贴近真实数据。完全吻合？行，就它了。做不到完全吻合？没关系，大差不差、足够接近就行。 If we can't figure out what the function looks like, stop forcing it. Say we have some data points of x and y and want their relationship. Enter humanity's most brilliant method ever: guessing. Start with an arbitrary line ŷ = w·x + b, then nudge w (the slope) and b (the intercept) bit by bit until the line hugs the data. Perfect fit? Great, done. Can't fit perfectly? No problem — close enough is good enough.

恭喜，你刚才做的事——「先随便猜、看误差、往误差变小的方向调」——就是所有现代 AI 训练的完整缩影。剩下的整个课程，本质上只是把「一条直线」升级成「一个巨大无比的函数」，把「手动拖滑块」升级成「自动调几千亿个滑块」而已。 Congratulations — what you just did ("guess, check the error, adjust toward less error") is the entire essence of modern AI training. The rest of this course is really just upgrading "a straight line" into "an enormous function," and "dragging sliders by hand" into "auto-tuning hundreds of billions of sliders."

2.1 激活函数：从线性到非线性2.1 Activation functions: from linear to nonlinear

如果数据本身弯弯曲曲，那不论怎么调 w 和 b，一条直线都无法贴近它。怎么办？很简单：在线性函数外面再套一层非线性运算——这就是激活函数。它的唯一目的，是把死气沉沉的线性关系「盘活」，变成表达能力更强的非线性关系。别被名字吓到，常用的激活函数都简单到爆炸： If the data itself is curvy, no amount of adjusting w and b will make a straight line fit. The fix is simple: wrap a nonlinear operation around the linear function — that's an activation function. Its entire purpose is to bring the lifeless linear relationship to life, giving it far more expressive power. Don't let the name scare you — the common ones are embarrassingly simple:

2.2 套娃：理论上能逼近任何函数2.2 Stack it: approximating (almost) anything

一层「线性变换 + 激活」弯得还不够灵活？那就把它整个当作输入，再来一层线性变换、再套一个激活函数——无限套娃下去。数学上可以证明：这样的结构理论上能逼近任意连续函数（万能逼近定理）。空口无凭，下面这个演示里有一个真的神经网络正在你的浏览器里训练： One round of "linear + activation" not bendy enough? Take the whole thing as input, apply another linear step, wrap another activation — and keep nesting, like Russian dolls. It can be proven mathematically that this structure can approximate any continuous function (the Universal Approximation Theorem). Don't take my word for it — the demo below trains a real neural network right inside your browser:

2.3 画成图：这才叫「神经网络」2.3 Draw it: now it looks like a "neural network"

公式套两层人就头大了，所以我们换一种更直观的画法：每个变量画成一个圈（神经元），圈与圈的连线代表一次「乘以 w 再相加」。左边是输入层，右边是输出层，中间被包在复杂变换里、从外面看不到的叫隐藏层。这一大坨互相连接的圈，就是神经网络。 Formulas get headache-inducing after two layers of nesting, so we switch to a friendlier picture: draw each variable as a circle (a neuron); each connecting line means "multiply by a w and add up." The left side is the input layer, the right side the output layer, and the circles buried inside — invisible from outside — form the hidden layers. This whole tangle of connected circles is a neural network.

每层神经元的个数可以无限加，隐藏层的层数也可以无限加，于是就能构成极其复杂的非线性函数。函数虽复杂，目标却始终简单：根据已知的一组 x 和 y，猜出所有的 w 和 b 该是多少。参数只有两个时靠肉眼能猜，现在有成千上万个参数怎么办？这就是下一章的主角。 You can add as many neurons per layer and as many hidden layers as you like, building an outrageously complex nonlinear function. But however complex the function, the goal stays simple: given known pairs of x and y, figure out what all the w's and b's should be. With two parameters you can eyeball it — with thousands upon thousands, you can't. That's the hero of the next chapter.

3.1 损失函数：给「拟合得好不好」打分3.1 The loss function: scoring "how good is the fit"

什么样的 w 和 b 算「好」？直觉上，就是预测值 ŷ 离真实值 y 越近越好。把每个点的误差 (ŷ−y) 平方后取平均——平方一来解决了绝对值不平滑、要分类讨论的麻烦，二来放大了大误差的惩罚——这就是均方误差 MSE，最常用的一种损失函数（Loss Function）： What makes a w and b "good"? Intuitively: predictions ŷ as close to the real y as possible. Square each error (ŷ−y) and average — squaring avoids the non-smooth, case-splitting hassle of absolute values, and punishes big errors extra hard. This is Mean Squared Error (MSE), the most common kind of loss function:

简单问题（比如线性回归）可以直接令导数等于 0，一步解出最小值。但神经网络的损失函数复杂到变态，解不出来。人们的办法简单粗暴：一点点试。看看 w 增大一点，L 是变大还是变小；变小就继续，变大就反着来。「w 变化一点点，L 会变化多少」——这正是导数（多个参数时叫偏导数，它们排成的向量叫梯度 Gradient）。沿着梯度反方向走，就是下降最快的方向。 For simple problems (like linear regression) you can set the derivative to zero and solve for the minimum in one shot. A neural network's loss is monstrously complex — no closed-form solution. So we do something crude but effective: try, a little at a time. Increase w slightly — did L go up or down? If down, keep going; if up, reverse. "How much does L change when w changes a tiny bit" is exactly a derivative (with many parameters, a partial derivative; the vector of all of them is the gradient). Walking against the gradient is the fastest way downhill.

3.2 反向传播：链式法则 = 齿轮咬合3.2 Backpropagation: the chain rule is a gear train

神经网络整体复杂，但层与层之间的关系非常简单。想知道第一层的 w₁ 变化一点会让 L 变多少？拆开看：w₁ 动一点 → 隐藏层 a 动多少 → 输出 ŷ 动多少 → 损失 L 动多少。每一环都是一个简单的偏导数，乘起来就是答案。就像一串咬合的齿轮：第一个转一圈，最后一个转几圈？把每级传动比乘起来就行。这就是微积分里的链式法则。 The network as a whole is complex, but the relationship between adjacent layers is dead simple. Want to know how much L changes when the first layer's w₁ wiggles? Break it down: w₁ wiggles → how much does hidden a move → how much does output ŷ move → how much does loss L move. Each link is one simple partial derivative; multiply them together and you have the answer. It's a gear train: turn the first gear once — how far does the last one turn? Multiply the gear ratios. That's the chain rule from calculus.

数据背后的真实规律往往很简单，但如果模型太复杂，它会连数据里的噪声和随机波动一起学会。在训练数据上表现完美、在新数据上表现糟糕，这叫过拟合（Overfitting）；模型在没见过的数据上的表现能力，叫泛化能力（Generalization）。亲手制造一次过拟合，感受一下： The true pattern behind data is often simple — but an overly complex model will learn the noise and random wobble along with it. Performing perfectly on training data while failing on new data is called overfitting; a model's ability to handle unseen data is its generalization. Let's manufacture some overfitting with our own hands:

对抗过拟合的武器库The anti-overfitting arsenal

人们为了对抗过拟合绞尽了脑汁，思路分三路——改数据、改模型、改训练过程： People have racked their brains against overfitting along three lines — change the data, change the model, or change the training process:

5.1 先把公式写优雅：矩阵5.1 First, make the math elegant: matrices

输入多了、神经元多了，公式里的 w₁₁x₁ + w₁₂x₂ + … 密密麻麻毫无美感。把它们打包成矩阵乘法，每一层就浓缩成一行： With more inputs and more neurons, formulas like w₁₁x₁ + w₁₂x₂ + … turn into an unreadable wall. Pack them into a matrix multiplication and each layer condenses to one line:

我们之前默认「每个神经元与前一层所有神经元相连」，这种结构叫全连接层。它在图像上有两个大问题：① 参数爆炸——一张 30×30 的小灰度图平铺开是 900 个输入，接 1000 个神经元就是 90 万个参数；② 平铺摧毁了像素间的空间关系，图片平移一下，所有输入全变了。 So far we've assumed every neuron connects to all neurons in the previous layer — a fully-connected layer. For images it has two big problems: ① parameter explosion — a tiny 30×30 grayscale image flattens into 900 inputs, and connecting them to 1,000 neurons costs 900,000 parameters; ② flattening destroys the spatial relationships between pixels — shift the image slightly and every input changes.

5.2 卷积：拿着放大镜扫过图片5.2 Convolution: sweeping a magnifying glass across the image

解决办法：不再全连接，而是拿一个小小的 3×3 矩阵（卷积核）在图片上滑动。每停一处，把卷积核和它盖住的 3×3 像素对应相乘再求和，得到输出图的一个像素。这就是卷积运算。在传统图像处理（PS 滤镜）里卷积核是人手工设计的；在深度学习里，卷积核里的数值是训练出来的——网络自己学会该「看」什么特征。 The fix: drop full connectivity and slide a tiny 3×3 matrix (a kernel) across the image. At each stop, multiply the kernel with the 3×3 pixels it covers, element-wise, then sum — that's one pixel of the output. This is convolution. In classic image processing (think Photoshop filters), kernels are designed by hand; in deep learning, the kernel's values are learned — the network figures out for itself what features to "look" for.

用卷积层替换全连接层，一举两得：参数从几十万骤降到卷积核的区区几个，还天然保留了局部空间结构。再配上给特征图降采样、保留主要特征的池化层（Pooling），「卷积→池化→卷积→池化→全连接」堆起来，就是大名鼎鼎的卷积神经网络 CNN——手写数字识别、人脸识别背后都是它。 Swapping fully-connected layers for convolutions wins twice: parameters plummet from hundreds of thousands to the kernel's handful, and local spatial structure is preserved for free. Add pooling layers that downsample feature maps while keeping the essentials, stack "conv → pool → conv → pool → fully-connected," and you have the celebrated Convolutional Neural Network (CNN) — the engine behind handwritten-digit and face recognition.

6.1 Token 与编码：两种极端都不行6.1 Tokens and encoding: both extremes fail

先把句子切成最小单位——token（词或子词），这一步叫分词（tokenization）。然后怎么编码成数字？两种极端方案都不行：只用一个编号（我=1，你=2，地球=3568）——维度太低，编号本身毫无语义，「猫」和「狗」的编号看不出任何亲缘；one-hot 独热编码——准备一个词表那么长的向量，每个词只有一个位置是 1——维度太高太稀疏，且任意两个词的向量都互相垂直，依然算不出相关性。 First, chop sentences into minimal units — tokens (words or subwords); that step is tokenization. Then how to encode them as numbers? Both extreme options fail. A single ID (I=1, you=2, Earth=3568): dimension too low, the number itself carries zero meaning — the IDs of "cat" and "dog" reveal no kinship. One-hot encoding — a vector as long as the whole vocabulary, with a single 1 per word: dimension too high and sparse, and every pair of vectors is perpendicular, so similarity still can't be computed.

解法是折中：词嵌入（Word Embedding）——给每个词一个维度不高不低（比如 300 维、几千维）的稠密向量，每个位置都可以看作某种「特征」。关键在于：这些数值不是人定义的，是训练出来的。具体特征是什么含义，可能人类完全无法理解——但没关系，好用就行。所有词向量摞成一个大矩阵，叫嵌入矩阵（Embedding Matrix）；两个词的相关性，用向量的点积或余弦相似度来衡量。这样，「语言的联系」第一次变成了可以用公式算的东西。 The answer is the middle path: word embeddings — give every word a dense vector of moderate dimension (say 300, or a few thousand), where each position can be read as some "feature." The crucial part: these values are not defined by humans; they are learned. What each feature means may be utterly incomprehensible to us — and that's fine, as long as it works. Stack all the word vectors into one big matrix — the embedding matrix. Measure two words' relatedness with the dot product or cosine similarity of their vectors. For the first time, "how words relate" became something you can calculate with a formula.

6.2 RNN：给网络装上「记忆」6.2 RNNs: giving the network a memory

有了词向量，还有个问题：普通神经网络一股脑接收整句话，既撑爆输入层，又体现不了词的先后顺序。循环神经网络（RNN）的思路：一次只喂一个词，但让每个词算完后输出一个隐藏状态 h，和下一个词一起参与运算——前面所有词的信息就这样一站一站往后传，像接力棒一样。 Embeddings in hand, one problem remains: an ordinary network swallowing a whole sentence at once both bloats the input layer and ignores word order. The Recurrent Neural Network (RNN) takes another route: feed one word at a time, but have each step emit a hidden state h that joins the next word's computation — information from all earlier words relays forward, station by station, like a baton.

7.0 先鸟瞰：整台机器只分三块7.0 Bird's-eye view: the machine has just three parts

别怕架构图，Transformer 整体上就三大块。① 嵌入（Embedding）：把每个 token 变成向量，并加上位置信息——上一章刚学过；② 若干个 Transformer 块：通过「注意力机制」，让每个词向量把其他词的信息按需「加」进自己——彼此包含一下对方；③ 输出概率（Probabilities）：把最终向量映射成词表里每个词的分数，换算成「下一个词」的概率分布。任务本身简单明确：输入一串文字，预测下一个词。剩下的一切都只是「拆解」——把一个训练不动的巨型映射，拆成一节节容易训练的小环节。 Don't fear the architecture diagram — the Transformer has just three parts. ① Embedding: turn each token into a vector and add position info (last chapter's material); ② a stack of Transformer blocks: via the "attention mechanism," each word vector mixes in information from other words as needed — everyone absorbing a bit of everyone; ③ Probabilities: map the final vectors to a score for every word in the vocabulary, converted into a probability distribution over "the next word." The task itself is simple and crisp: given a string of text, predict the next token. Everything else is decomposition — splitting one untrainable giant mapping into small, trainable stages.

7.1 位置编码：给每个词一个「座位号」7.1 Positional encoding: giving every word a seat number

注意力机制会让所有词同时互相观察——但这样一来，「我打你」和「你打我」就没区别了！所以在进入注意力之前，先给每个词向量加上一个位置编码（Positional Encoding）：一个只由位置决定的向量。原始论文用不同频率的 sin/cos 函数生成它——低维度波动快、高维度波动慢，就像用多个「钟表指针」组合出每个位置独一无二的签名。 Attention lets all words observe each other simultaneously — but then "dog bites man" and "man bites dog" would look identical! So before attention, each word vector gets a positional encoding added: a vector determined purely by position. The original paper generates it with sin/cos waves of different frequencies — fast-oscillating in low dimensions, slow in high ones — like combining many "clock hands" into a unique signature for every seat.

7.2 注意力的直觉：「它」指的是谁？7.2 The intuition of attention: what does "it" refer to?

在讲数学之前，先感受注意力在解决什么问题。看这句话：「动物没有过马路，因为它太累了」——你不假思索就知道「它」指动物。但如果结尾换成「因为它太宽了」，「它」就指马路了。同一个词，该关注谁，取决于整句话的语境。注意力机制就是让每个词自动算出「我该按什么比例关注其他词」。 Before the math, feel the problem attention solves. Read: "The animal didn't cross the road because it was too tired" — you instantly know "it" means the animal. But swap the ending to "because it was too wide," and "it" now means the road. The same word should attend to different words depending on context. Attention lets every word automatically compute "in what proportions should I focus on the other words."

7.3 注意力的机关：Q、K、V 分步计算器7.3 The machinery: a step-by-step Q, K, V calculator

那「按什么比例关注」是怎么算出来的？每个词向量会通过三个可训练的矩阵 W^Q、W^K、W^V，变出三个分身：Q（Query，查询）——我在找什么；K（Key，键）——我能提供什么线索；V（Value，值）——我实际携带的信息。用我的 Q 和每个词的 K 做点积，就得到「我对每个词的关注度」；softmax 归一化成比例后，按比例把大家的 V 加权求和——就得到了一个融合了全句上下文的新向量。下面用一个真实的小例子（4 个词、真实数字）一步步算一遍： So how are those "proportions of focus" computed? Each word vector passes through three trainable matrices W^Q, W^K, W^V, producing three alter egos: Q (Query) — what am I looking for; K (Key) — what clue can I offer; V (Value) — the information I actually carry. Dot my Q with every word's K to get "how much I care about each word"; normalize with softmax into proportions; then take a weighted sum of everyone's V — and out comes a new vector infused with the whole sentence's context. Let's compute a real miniature example (4 tokens, real numbers), one step at a time:

7.4 多头注意力：多几双眼睛看关系7.4 Multi-head attention: several pairs of eyes on the sentence

词与词的关系不止一种视角：语法上的、指代上的、语义上的……如果只算一次注意力，灵活性就差了。于是把 Q、K、V 各自拆成多组，每组独立做一遍注意力（每组叫一个头 head），各头结果拼接后再过一次线性变换。每个头自然分工出不同的「观察角度」： Words relate in more than one way: grammar, coreference, meaning… computing attention once limits flexibility. So Q, K, V are split into several groups, each running attention independently (each group is a head); the heads' outputs are concatenated and passed through one more linear map. Different heads naturally specialize in different "ways of looking":

7.5 因果掩码：不许偷看答案7.5 The causal mask: no peeking at the answer

生成文字时是一个词一个词往外蹦的——预测第 3 个词时，模型不可能看到第 4、5 个词。为了让训练环境和真实生成一致，训练时用一个掩码（mask）把「未来的词」遮住：注意力分数矩阵的右上三角全部设为 −∞，softmax 之后这些位置的权重变成 0。 Text is generated one token at a time — when predicting token 3, the model cannot see tokens 4 and 5. To make training match real generation, a mask hides "future" tokens during training: the upper-right triangle of the score matrix is set to −∞, so after softmax those weights become exactly 0.

7.6 组装整机：完整架构图7.6 Assembling the machine: the full architecture

核心零件已集齐，剩下的都是老朋友：残差连接（Add）让信号抄近道、防止深层网络梯度消失；归一化（Norm）让数值分布稳定；前馈网络（Feed Forward）就是第 2 章的经典小神经网络。原论文是「编码器-解码器」结构（为翻译设计）：编码器读入原文，解码器一边看编码器的输出（作为 K、V），一边逐词生成译文。点击 / 触摸任何模块查看说明；再点「GPT 视角」——你会发现 GPT 只是把左半边整个扔掉了。 All core parts collected — the rest are old friends: residual connections (Add) give signals a shortcut and fight vanishing gradients in deep stacks; normalization (Norm) keeps value distributions stable; the feed-forward network is just Chapter 2's classic little neural net. The original paper uses an encoder–decoder design (built for translation): the encoder reads the source text; the decoder generates the translation word by word while consulting the encoder's output (as its K and V). Click / tap any block for its explanation — then hit "GPT view" and discover that GPT simply threw the entire left half away.

7.7 出口：从分数到下一个词7.7 The exit: from scores to the next word

走完所有 Transformer 块后，最后一层线性变换把向量投射到词表大小的分数（logits），softmax 把分数变成概率。之后并非总是选概率最高的词——那样太死板。温度（temperature）控制概率分布的「尖锐程度」，top-k 限定只在概率最高的 k 个词里抽签。选出的词接回输入末尾，再算一轮——如此循环，文字就一个个「蹦」出来了。这就是自回归生成。 After the last Transformer block, a final linear layer projects the vector to vocabulary-size scores (logits), and softmax turns scores into probabilities. The model doesn't always pick the top word — too rigid. Temperature controls how "sharp" the distribution is; top-k restricts the lottery to the k most likely words. The chosen word is appended to the input and the whole thing runs again — around and around, tokens popping out one by one. That's autoregressive generation.