AI/ML Deep Dive: Pre-training and Fine-tuning

Imagine handing a new hire the entire internet and telling them to read it through — doing exactly one thing: look at the prefix, guess the next word. Right or wrong isn't judged morally; the gradient just tunes their intuition for the statistical patterns of text.

If we trained a fresh model for every task (translation, QA, code generation), each one would have to relearn what English is, what a sentence is, what facts look like — economically and data-wise, impossible. Pre-training packages "language + common sense" into a shared foundation once, and downstream tasks only need lightweight specialization on top. This is the entire premise of foundation models.

The core task is brutally simple: given the first N tokens, predict token N+1. Compare the prediction with the real token, take the gradient, repeat trillions of times.

# Pseudocode: one pre-training step
text = sample_from_internet()              # a stew of web pages, books, code, papers
tokens = tokenize(text)                    # slice into a token sequence
for i in range(len(tokens) - 1):
    pred = model(tokens[:i+1])             # probability distribution over the next token
    loss = cross_entropy(pred, tokens[i+1])# how wrong was the guess
    loss.backward(); optimizer.step()      # nudge parameters a little

Scale dictates capability: as data grows from GB to TB and parameters from hundreds of millions to hundreds of billions, "emergent" abilities like reasoning and code generation appear.

# Feel the pre-training objective via Hugging Face: get the next-token distribution at any position
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tok("The capital of France is", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits        # [1, seq_len, vocab_size]
next_id = logits[0, -1].argmax()           # pick the most likely next token
print(tok.decode(next_id))                  # " Paris"

• Karpathy: "Let's build GPT" on YouTube (2 hours of hand-coding a mini GPT — the best intro)
• Hugging Face: "How does a language model work" chapter

Canonical: GPT-2/3 and the Llama base models are "pre-training only" snapshots.
Closer to home: when you see llama-3-8b (no instruct suffix), that's the raw pre-trained model — for completion-style tasks like extending an investment memo or generating training data, it's often more useful than the chat-tuned version.

Like handing the new grad a binder of "model answers" and having them copy each one out a thousand times — turning a model that "can continue text" into a model that "follows instructions."

A pre-trained model only continues text. It doesn't know that "please translate this passage" should produce a translation and not commentary. SFT uses curated instruction-response pairs to nudge the model's output onto a track of "obedient, useful, well-formatted." This is the step that turns a base model into an instruct model — the entry point for every ChatGPT- or Claude-style product.

The training objective is identical to pre-training (predict the next token). Only the data changes: from "internet stew" to "high-quality instruction-response pairs."

# A sample looks like this:
{
  "instruction": "Explain caching in one sentence",
  "response":   "Caching stores the result of an expensive computation so it can be reused next time."
}
# Concatenated at training time:
prompt = "<|user|>Explain caching in one sentence<|assistant|>"
target = prompt + response
# Key detail: loss is computed only on the response portion, not the prompt
loss = cross_entropy(model(target), target, mask=is_response_token)

Usually 10K to 1M carefully chosen samples are enough to transform the model's behavior. Quality >> quantity.

# The simplest possible SFT with trl, under 20 lines
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.2-1B"
tok   = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
ds    = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1000]")

trainer = SFTTrainer(
    model=model, tokenizer=tok, train_dataset=ds,
    args=SFTConfig(output_dir="sft-llama", num_train_epochs=1,
                   per_device_train_batch_size=2, learning_rate=2e-5),
)
trainer.train()                            # after this, the model "listens"

• Hugging Face TRL docs: huggingface.co/docs/trl (high-level APIs for SFT/DPO/PPO)
• Sebastian Raschka blog: "Practical Tips for Fine-tuning LLMs"

Canonical: InstructGPT, Alpaca, Llama-3-Instruct.
Closer to home: SFT a "parenting helper" — feed 500 samples of "parent question → answer in the style you actually endorse," and the model will start replying with the tone and depth you want instead of generic platitudes.

Like an App Store rating system: let humans label "which of these two answers is better," train a "judge model" on those preferences, then let the main model grind for the judge's approval — gradually learning to produce outputs humans prefer.

SFT teaches the model to imitate written examples, but many qualities resist a canonical answer: what counts as "polite but not verbose"? "Helpful but not overreaching"? Writing examples by hand is slow and inconsistent, while comparing two answers to pick the better one is easy. RLHF turns a "hard to demonstrate" problem into an "easy to choose" problem — the key step that made ChatGPT feel "actually thoughtful" to mainstream users.

Three independent training stages:

# Step 1: SFT model already exists (previous card)

# Step 2: train a Reward Model (RM)
for (prompt, response_A, response_B, human_pref) in pref_data:
    score_A = reward_model(prompt, response_A)
    score_B = reward_model(prompt, response_B)
    # Push the human-preferred answer's score higher
    loss = -log(sigmoid(score_chosen - score_rejected))

# Step 3: use PPO to let the main model maximize the RM's score
for prompt in prompts:
    response = sft_model.generate(prompt)
    reward   = reward_model(prompt, response)        # judge scores it
    # PPO: move toward higher reward but stay close to SFT (KL penalty)
    sft_model.ppo_step(reward, kl_penalty=0.1)

The KL penalty matters — without it, the model discovers "reward hacking" and emits weird phrasings that the RM loves but no human does.

# trl's PPOTrainer wraps all three stages (here, just the step-3 skeleton)
from trl import PPOTrainer, PPOConfig
from transformers import pipeline

reward = pipeline("text-classification", model="my-reward-model")
ppo = PPOTrainer(config=PPOConfig(batch_size=8),
                 model=sft_model, tokenizer=tok)

for batch in dataloader:
    responses = ppo.generate(batch["prompt"])
    # RM scores each (higher = more aligned with human preference)
    scores = [reward(p + r)[0]["score"] for p, r in zip(batch["prompt"], responses)]
    ppo.step(batch["prompt"], responses, scores)

• Hugging Face: "Illustrating RLHF" (the clearest visual intro)
• Chip Huyen: "RLHF: Reinforcement Learning from Human Feedback"

Canonical: ChatGPT, Claude, and Gemini all use RLHF or variants.
Closer to home: add thumbs-up / thumbs-down buttons to your team's AI assistant, accumulate enough preference data, train an RM, and PPO-optimize the main model — this is the standard recipe for crystallizing internal team preferences into the model itself.

It's RLHF collapsed into one SQL: skip the "train a judge + run PPO" two-step and tune the model directly from preference pairs — no reward model, no reinforcement learning, training as stable as SFT.

RLHF has three pain points: (1) you train two models (RM + policy), doubling machines and engineering; (2) PPO is finicky, unstable, prone to collapse; (3) the RM is an approximation layer that introduces bias. DPO (Stanford, 2023) proves that under certain assumptions, "maximizing reward" is equivalent to a very simple classification loss, letting you bypass the RM and RL entirely and train directly on preference pairs. Result: training cost halved, stability dramatically improved.

Core idea: for every preference pair, raise the chosen response's probability relative to a reference model, and lower the rejected one's.

# Training data: (prompt, chosen, rejected) triples
for (prompt, chosen, rejected) in data:
    # new model's log-prob for chosen minus reference (SFT) model's log-prob for chosen
    log_ratio_chosen   = log P_new(chosen|prompt)   - log P_ref(chosen|prompt)
    log_ratio_rejected = log P_new(rejected|prompt) - log P_ref(rejected|prompt)
    # Make chosen's lift > rejected's lift
    loss = -log(sigmoid(beta * (log_ratio_chosen - log_ratio_rejected)))

In plain English: the new model, compared to the SFT reference, should prefer chosen more and rejected less. That direct, no RM, no PPO, no sampling, no reward hacking headaches.

# trl's DPOTrainer — the interface is as simple as SFTTrainer
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

# Preference data: each row has prompt / chosen / rejected
ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model=sft_model,                       # the SFT model from the previous step
    ref_model=sft_model,                   # reference model (frozen)
    train_dataset=ds, tokenizer=tok,
    args=DPOConfig(beta=0.1, num_train_epochs=1,
                   learning_rate=5e-7, output_dir="dpo-llama"),
)
trainer.train()                            # done — quality roughly matches RLHF at far less effort

• Friendly walkthrough of the paper: HF Blog "Preference Tuning LLMs"
• Sebastian Raschka: "LLM Training: RLHF and Its Alternatives"

Canonical: Llama-3-Instruct, Zephyr, the Tulu series all use DPO.
Closer to home: aligning a research-analyst assistant — collect 2,000 samples of "two answers to the same question + which one is more professional," DPO-tune, finish on one GPU in a few hours, and the quality often matches a full RLHF pipeline.