AI/ML Deep Dive: LLM Foundations

Trade the "read one word at a time" serial pipeline for a "whole-team-in-one-meeting" parallel architecture — every token sees the full text at once before deciding what to say.

Before 2017, RNN/LSTM dominated text processing. They were essentially a for-loop: you couldn't compute token t+1 until you finished token t. Two pain points followed: (1) training couldn't be parallelized, so it was slow; (2) long-range dependencies tended to fade away (vanishing gradients). The Transformer uses self-attention to directly connect any two positions in one shot — GPUs run at full utilization in parallel, and long-range signal no longer gets lost.

Picture each token as an object with three properties: Query (what I'm looking for), Key (what I can offer), Value (what I actually carry). Each layer roughly does this:

# Pseudocode: one Transformer block
for token in sequence:
    # 1. Score Q against every Key in the sequence (like a database JOIN)
    scores = token.Q @ all_tokens.K.T
    weights = softmax(scores)
    # 2. Pull others' Value vectors back, weighted
    token.new = weights @ all_tokens.V
    # 3. Pass through a feed-forward layer (a nonlinear projection)
    token.out = FFN(token.new + token)  # residual connection

Stack 12 to 96 of these blocks. Each layer lets every token "re-ask the whole sequence," and the representation gets richer with every pass.

# Load a Transformer with the transformers library — the whole picture in 10 lines
from transformers import AutoTokenizer, AutoModel
import torch

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

inputs = tok("Transformers changed NLP.", return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)

# out.last_hidden_state shape: [batch, seq_len, hidden_dim]
# Each token is now a 768-dim vector encoding its meaning in context
print(out.last_hidden_state.shape)  # torch.Size([1, 6, 768])

• Jay Alammar — The Illustrated Transformer (the classic visual intro)
• Karpathy — Let's build GPT from scratch (2-hour live coding)

Canonical: every modern LLM (GPT, Claude, Gemini, Llama) is a Transformer. Your day-to-day: when you paste a 5,000-word meeting transcript into Claude and ask for a summary, the Transformer's parallel attention is what lets the model scan the whole document in one pass instead of word by word.

Like a SQL JOIN — every word carries its own "query," matches it against the full text, finds the most relevant terms, and pulls a weighted copy of their content back.

"The cat sat on the mat because it was warm." Does "it" refer to the mat or the cat? Traditional models could only guess by proximity. Attention lets the position holding "it" actively poll the whole sentence: who is most relevant to me? It then pulls in the mat's vector with high weight, baking "it = mat" into its own representation. The essence: solving long-range coreference and contextual focus.

The core formula: Attention(Q,K,V) = softmax(Q·Kᵀ / √d) · V. In plain English: take dot products between Q and every Key to get similarity, normalize into weights, then take a weighted sum of all Values. A weighted lookup, nothing more.

# Attention in 4 lines of numpy
import numpy as np
def attention(Q, K, V):
    scores = Q @ K.T / np.sqrt(K.shape[-1])  # similarity
    weights = np.exp(scores) / np.exp(scores).sum(axis=-1, keepdims=True)
    return weights @ V                        # weighted sum

Multi-Head Attention = running several independent query channels in parallel, each head focusing on a different dimension of relation (syntax, semantics, coreference, etc.).

# See what GPT-2 is actually attending to
from transformers import GPT2Tokenizer, GPT2Model
tok = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2Model.from_pretrained("gpt2", output_attentions=True)

inputs = tok("The cat sat on the mat", return_tensors="pt")
out = model(**inputs)
# out.attentions[layer][batch, head, query_pos, key_pos]
attn = out.attentions[-1][0, 0]   # last layer, head 0
print(attn.shape)  # [6, 6] — attention weights from each token to every other

• Illustrated Transformer — Attention section
• Distill — Attention and Augmented RNNs

Canonical: in machine translation, each target word uses attention to look at the corresponding source words. Your day-to-day: when you read a bedtime story to your kid through Claude, the reason the model remembers that "little fox" from paragraph 1 is still the same fox in paragraph 3 is attention building coreference links across paragraphs.

Like a compiler lexing source code into tokens, an LLM must first slice a character stream into chunks and map each chunk to an integer ID. The model never sees characters — only IDs.

Vocabularies can't be too big (one ID per word → millions of entries → exploding matrices) nor too small (one ID per letter → sequences too long, semantics lost). The fix: BPE / WordPiece / SentencePiece automatically learn a vocabulary of around 50K subwords from high-frequency byte combinations. Common words stay whole; rare words get split into pieces.

BPE training is greedy compression: start from single characters, repeatedly merge the most frequent adjacent pair into a new token, and stop when the vocabulary hits target size.

# BPE intuition
# Start: ['l','o','w','e','r']
# 'l'+'o' is most frequent → merge → ['lo','w','e','r']
# 'lo'+'w' is most frequent → merge → ['low','e','r']
# High-frequency whole words survive; rare words break into subword pieces
# "tokenization" might become ["token", "ization"]

Result: in English, 1 token ≈ 4 characters ≈ 0.75 words; in Chinese, one character often costs 2-3 tokens. This directly drives your bill and your context budget.

import tiktoken  # OpenAI's official tokenizer

enc = tiktoken.get_encoding("cl100k_base")  # used by GPT-4

text_en = "Tokenization is fun!"
text_zh = "分词其实很有意思！"

print(enc.encode(text_en))         # [3404, 2065, 374, 2523, 0]
print(len(enc.encode(text_en)))    # 5 tokens
print(len(enc.encode(text_zh)))    # 11 tokens (Chinese is pricier!)

# Decode back
print(enc.decode([3404, 2065]))   # "Tokeniz" — not a complete word

• OpenAI Tokenizer playground (paste text and watch the split)
• Karpathy — Let's build the GPT Tokenizer

Canonical: evaluating API cost (OpenAI bills by token). Your day-to-day: the same meeting transcript costs 2-3x more in Chinese than in English. For high-frequency automation tasks (daily news summarization), writing the prompt in English while letting the model reply in Chinese saves money.

Attention is an unordered-set operation (like a Python set) — it has no idea which token comes first. Positional encoding stamps each token with a "position ID" so the model can tell "dog bites man" from "man bites dog."

The Q·Kᵀ computation in self-attention is completely permutation-invariant — shuffle the sentence, recompute, and you get the same answer. Obviously unacceptable. The fix: inject position information into the token embeddings so position 0 and position 5 start from different vectors.

Three generations of design:

• Sinusoidal (original Transformer): use sin/cos at different frequencies to generate a unique vector per position, added to the token embedding. Upside: can extrapolate to lengths never seen in training.
• Learned (GPT-2 / BERT): simply learn a [max_len, dim] lookup matrix. Simple, but no extrapolation.
• RoPE — Rotary Position Embedding (Llama / Qwen / today's standard): turn position into a rotation angle. Each token rotates by a position-dependent angle in a high-dimensional space, so the Q·K dot product naturally encodes relative distance. This is the de facto standard for long-context models.

# Intuition: RoPE rotates position-m vectors by m·θ
def rope(x, pos):
    # x is the token's Q or K vector, processed in pairs
    for i, (a, b) in enumerate(pairs(x)):
        theta = pos / 10000 ** (2*i / dim)
        a_new = a * cos(theta) - b * sin(theta)
        b_new = a * sin(theta) + b * cos(theta)
    return rotated_x

# Watch how positional encoding changes the output
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

# Same tokens, different positions → different hidden states
ids_a = tok("dog bites man", return_tensors="pt").input_ids
ids_b = tok("man bites dog", return_tensors="pt").input_ids

with torch.no_grad():
    h_a = model.transformer(ids_a).last_hidden_state
    h_b = model.transformer(ids_b).last_hidden_state

# Same three tokens, but the representations diverge because of position
print(torch.allclose(h_a, h_b))   # False

• HuggingFace — Designing Positional Encoding
• EleutherAI — Rotary Embeddings explained

Canonical: any order-sensitive task (translation, code generation, reasoning chains). Your day-to-day: when you ask Claude to analyze a 50-page PDF report, the model has to keep page 3 distinct from page 47 — that's positional encoding (RoPE) maintaining positional awareness across a long context.