Advanced

15 min read

#generative ai#Guide

Post-Training & Alignment for Generative AI

Comprehensive guide on Post-Training & Alignment for Generative AI.

Post-Training & Alignment for Generative AI

Pretraining teaches a language model to model patterns in large-scale token sequences.

But a pretrained model is not automatically a good assistant.

A base model may be able to continue text while still being poor at:

following explicit instructions
answering in a useful format
refusing unsafe requests
following system policies
using tools correctly
communicating uncertainty
adapting responses to users
producing preferred styles
avoiding undesirable behaviors

Post-training is the collection of techniques used to shape a pretrained model into a model that behaves according to a desired specification.

A simplified lifecycle is:

Architecture & Data Flow
Large-scale pretraining
 |
 v
 Base model
 |
 v
 Instruction tuning
 |
 v
 Preference / alignment training
 |
 v
 Safety + capability evaluation
 |
 v
 Assistant

The central idea is:

Pretraining builds broad language and world-modeling capability; post-training shapes how that capability is expressed and used.

This notebook covers:

supervised fine-tuning
instruction tuning
chat templates
preference data
reward modeling
RLHF
PPO-style optimization
DPO
preference optimization
rejection sampling
Constitutional AI concepts
safety alignment
tool-use training
structured-output training
continual post-training
alignment evaluation
reward hacking
distribution shift
alignment tax
production post-training pipelines
educational AI examples

Learning Objectives

By the end of this notebook, you should be able to:

Explain why pretrained models require post-training.
Distinguish pretraining from supervised fine-tuning.
Understand instruction-tuning datasets and chat templates.
Explain response masking and loss calculation for assistant turns.
Understand preference datasets.
Explain reward models at a conceptual level.
Understand the RLHF pipeline.
Explain PPO-style policy optimization at a high level.
Understand Direct Preference Optimization.
Compare SFT, RLHF, DPO, and rejection sampling.
Understand alignment objectives and safety training.
Explain reward hacking and proxy optimization.
Understand the alignment tax.
Train models for tool use and structured outputs.
Design evaluation suites for aligned assistants.
Build an educational AI post-training pipeline.
Identify common post-training failure modes.
Design a production post-training lifecycle.

1. Why Post-Training Exists

A pretrained model learns statistical patterns.

For example, given:

text
User:
What is machine learning?

Assistant:

a base model may continue with a plausible paragraph.

But a production assistant needs more predictable behavior:

text
User:
Explain machine learning to a beginner in 100 words.

Assistant:
Machine learning is a method...

The difference is not simply knowledge.

It includes:

instruction following
response style
task prioritization
safety behavior
formatting
conversational behavior

Post-training teaches these behaviors.

2. Base Model vs Instruction Model

A base model is optimized primarily for language modeling.

Architecture & Data Flow
Text
 |
 v
Next-token prediction
 |
 v
Base model

An instruction model is further trained on examples such as:

text
Instruction
 +
Expected response

Architecture & Data Flow
Prompt
 |
 v
Instruction-tuned model
 |
 v
Useful answer

This is why instruction-tuned models are generally much easier to use as assistants.

3. The Post-Training Stack

A modern post-training program may contain several stages:

Architecture & Data Flow
Base Model
 |
 v
Supervised Fine-Tuning
 |
 v
Preference Data
 |
 +--> Reward Modeling
 |
 +--> Direct Preference Optimization
 |
 +--> Other preference methods
 |
 v
Safety / Policy Training
 |
 v
Tool-use Training
 |
 v
Evaluation
 |
 v
Deployment

Not every model needs every stage.

4. Supervised Fine-Tuning

Supervised fine-tuning, commonly called SFT, trains a pretrained model on curated examples.

Example:

json
{
 "instruction": "Explain overfitting.",
 "response": "Overfitting occurs when..."
}

The model learns to produce the target response given the instruction.

Conceptually:

Architecture & Data Flow
Pretrained model
 +
Instruction dataset
 |
 v
SFT
 |
 v
Instruction model

5. SFT Objective

The underlying objective can remain token-level cross-entropy.

For target tokens:

[ L_{SFT}

-\sum_t \log P_\theta(y_t | x, y_{<t}) ]

where:

(x) is the prompt
(y) is the target response
(y_t) is the current target token

The key difference is the data distribution.

Instead of arbitrary internet text:

›instruction -> desired response

is emphasized.

6. Chat Templates

Modern chat models often represent conversations using structured message roles.

Example:


🐍 PythonInteractive WebAssembly
messages = [
 {
 "role": "system",
 "content": "You are a helpful tutor."
 },
 {
 "role": "user",
 "content": "Explain regression."
 },
 {
 "role": "assistant",
 "content": "Regression is..."
 }
]

A tokenizer or formatting layer converts this into the model's expected token sequence.

The exact template is model-specific.

7. Why Chat Templates Matter

A mismatch between training and inference formatting can reduce performance.

Suppose training uses:

text
<system>
<user>
<assistant>

while inference uses a different format.

The model may not behave as expected.

Therefore:

The tokenizer, chat template, special tokens, and post-training format should be treated as one system.

8. Response-Only Loss

In many instruction-tuning setups, the model is trained primarily on assistant tokens.

Conceptually:

Architecture & Data Flow
System tokens -> ignore
User tokens -> ignore
Assistant tokens -> calculate loss

Example:

text
User:
Explain gradient descent.

Assistant:
Gradient descent is an optimization algorithm...

The loss focuses on learning the desired assistant response.

This is often implemented using a label mask.

9. Example Loss Mask

Conceptually:

text
Tokens:

[USER] Explain regression [ASSISTANT] Regression is ...

Labels:

[-100] [-100] [-100] [-100] Regression is ...

In common PyTorch training conventions, -100 can be used to mark labels ignored by cross-entropy.

The exact masking strategy depends on the training framework.

10. High-Quality SFT Data

Strong SFT data should contain:

clear instructions
correct answers
useful formatting
representative tasks
realistic user requests
edge cases
appropriate difficulty
safety examples

Poor data can teach:

verbosity
incorrect facts
strange formatting
undesirable refusal behavior
inconsistent tone

SFT quality often matters more than simply increasing example count.

11. Instruction Data Categories

A broad assistant dataset might contain:

text
Question answering
Summarization
Classification
Extraction
Transformation
Writing
Coding
Reasoning-oriented tasks
Structured outputs
Tool calls
Safety

For specialized assistants, the mixture should match the product.

12. SFT Data Sources

Possible sources include:

expert-written examples
synthetic examples
teacher-model outputs
human demonstrations
corrected production examples
tool-use traces
domain-specific datasets

The previous notebooks on synthetic data and distillation are directly relevant here.

Architecture & Data Flow
Synthetic Data
 |
 v
Quality filtering
 |
 v
SFT dataset
 |
 v
Post-training

13. Preference Data

SFT tells the model:

"This is a good response."

Preference data tells the model:

"Response A is better than Response B."

Example:

json
{
 "prompt": "Explain overfitting.",
 "chosen": "Overfitting occurs when...",
 "rejected": "Overfitting is when the model..."
}

Preference data is useful because many responses can be technically acceptable while differing in:

clarity
helpfulness
relevance
safety
style
completeness

14. Why Preference Learning Helps

Suppose both responses are correct.

text
A:
Clear, concise, beginner-friendly.

B:
Correct but unnecessarily complicated.

SFT can learn from A.

Preference training can explicitly teach:

›A > B

This helps encode relative quality.

15. Pairwise Preferences

A preference dataset can contain:

text
Prompt
Candidate A
Candidate B
Preference

The preference can be:

text
A preferred
B preferred
tie

More candidates can also be ranked:

›A > C > B > D

which can be converted into pairwise comparisons or other preference-learning formats.

16. Sources of Preference Labels

Preference labels can come from:

humans
domain experts
teacher models
reward models
rules
hybrid systems

For high-value alignment work, human judgments remain important because automated judges can share systematic biases with the model being trained.

17. Reward Models

A reward model attempts to predict human or target preferences.

Architecture & Data Flow
Prompt
 |
 +--> Response A --> Reward Model --> 0.91
 |
 +--> Response B --> Reward Model --> 0.37

The reward model learns from preference comparisons.

Conceptually:

Architecture & Data Flow
Human preferences
 |
 v
Preference dataset
 |
 v
Reward model
 |
 v
Score candidate responses

18. Reward Model Training

Suppose:

›Response A preferred over B

The reward model should learn:

[ r(A) > r(B) ]

A common conceptual objective is a pairwise logistic loss:

[ L = -\log \sigma(r_A-r_B) ]

where:

(r_A) is the reward for A
(r_B) is the reward for B
(\sigma) is the sigmoid function

19. Reward Models Are Proxies

A reward model does not directly represent "true helpfulness."

It approximates a preference signal.

This creates an important principle:

Optimizing a proxy is not the same as optimizing the underlying goal.

For example, a reward model may learn to favor:

longer responses
polite language
certain formatting
confident wording

even when these do not actually improve the answer.

20. Reward Hacking

Reward hacking occurs when the policy discovers ways to increase the measured reward without genuinely improving the desired behavior.

Example:

text
Goal:
Helpful answer

Reward model:
Prefers longer answers

Model learns:
Make every answer extremely long

Reward:

›high

Actual usefulness:

›low

This is a classic proxy-optimization problem.

21. RLHF

RLHF stands for:

Reinforcement Learning from Human Feedback.

A simplified pipeline:

Architecture & Data Flow
 Base Model
 |
 v
 Supervised Fine-Tuning
 |
 v
 SFT Model
 |
 v
 Generate Responses
 |
 v
 Human Preferences
 |
 v
 Reward Model
 |
 v
 Policy Optimization
 |
 v
 Aligned Model

22. RLHF Stage 1: SFT

Start with demonstrations:

›instruction -> high-quality response

Train an SFT model.

This provides a useful initialization for later preference optimization.

23. RLHF Stage 2: Preference Collection

Generate multiple responses:

Architecture & Data Flow
Prompt
 |
 +--> Response A
 +--> Response B
 +--> Response C

Human annotators compare them.

Example:

text
A > C
B > C
A > B

These rankings form preference data.

24. RLHF Stage 3: Reward Model

Train a model to predict preferences.

Architecture & Data Flow
Preference data
 |
 v
Reward model

Then:

Architecture & Data Flow
Prompt + response
 |
 v
Reward score

25. RLHF Stage 4: Policy Optimization

The language model becomes the policy.

Conceptually:

Architecture & Data Flow
Prompt
 |
 v
Policy
 |
 v
Response
 |
 v
Reward model
 |
 v
Reward
 |
 v
Policy update

The objective is to increase reward while constraining undesirable divergence from the starting model.

26. PPO-Style Optimization

PPO, Proximal Policy Optimization, became a widely discussed approach for RLHF.

A simplified idea is:

Architecture & Data Flow
Old policy
 |
 v
Generate response
 |
 v
Evaluate reward
 |
 v
Update policy
 |
 v
Keep update within a controlled region

The "proximal" constraint helps prevent excessively large policy updates.

27. Why Constrain the Policy?

Suppose the policy is aggressively optimized against the reward model.

It may move far from the SFT model and exploit weaknesses in the reward function.

A common conceptual objective therefore includes a divergence penalty.

text
Reward improvement
 -
Policy divergence penalty

The balance matters.

28. KL Divergence in RLHF

A KL penalty can discourage the new policy from moving too far from a reference model.

Conceptually:

[ L = Reward#

\beta D_{KL}(\pi_\theta || \pi_{ref}) ]

where:

(\pi_\theta) is the trainable policy
(\pi_{ref}) is the reference policy
(\beta) controls the penalty

The exact implementation differs across algorithms.

29. DPO

Direct Preference Optimization, or DPO, is a preference-learning method that avoids training a separate reward model and running a conventional online RL loop in the same way as RLHF.

The conceptual input is:

text
Prompt
Chosen response
Rejected response

The model is optimized directly using preference pairs.

Architecture & Data Flow
Preference dataset
 |
 v
DPO objective
 |
 v
Policy model

30. Why DPO Is Attractive

Compared with a full reward-model + RL pipeline, DPO can be:

simpler
easier to implement
easier to debug
more stable for some workloads
less operationally complex

But DPO is not automatically superior.

Performance depends on:

preference-data quality
base model
hyperparameters
task
evaluation

31. DPO Intuition

Suppose:

text
Prompt:
Explain regularization.

Chosen:
Regularization reduces effective model complexity...

Rejected:
Regularization always increases model complexity...

DPO pushes the model toward assigning higher relative likelihood to the preferred answer than the rejected answer, using a reference policy to stabilize the objective.

32. SFT vs DPO vs RLHF

Method	Main Signal	Main Complexity
SFT	Demonstrations	Dataset quality
DPO	Preferences	Preference quality
RLHF	Reward + preference	Reward model + RL infrastructure
Rejection Sampling	Best accepted outputs	Candidate generation + filtering

There is no universal winner.

33. Rejection Sampling

A simple alignment strategy:

Architecture & Data Flow
Prompt
 |
 v
Generate N responses
 |
 v
Score / filter
 |
 v
Keep best
 |
 v
Fine-tune

Example:

Architecture & Data Flow
10 candidates
 |
 v
quality judge
 |
 v
2 accepted
 |
 v
SFT

This can be an effective bridge between synthetic data and post-training.

34. Iterative Rejection Sampling

The process can repeat:

Architecture & Data Flow
Model v1
 |
 v
Generate candidates
 |
 v
Filter
 |
 v
SFT
 |
 v
Model v2
 |
 v
Generate better candidates

This creates iterative self-improvement.

However, evaluation and contamination controls are necessary to prevent feedback loops.

35. Constitutional AI Concepts

A model can be trained against explicit principles rather than relying entirely on individual human comparisons.

A simplified pattern:

Architecture & Data Flow
Response
 |
 v
Principle / constitution
 |
 v
Critique
 |
 v
Revision
 |
 v
Training example

Principles might specify:

avoid harmful instructions
protect privacy
be honest about uncertainty
follow authorized instructions
avoid discrimination

The exact methodology varies.

36. Safety Alignment

Safety post-training can teach the model to:

refuse unsafe requests
avoid disallowed content
protect sensitive information
resist prompt manipulation
respect system policies
safely handle uncertainty

But safety training should not simply maximize refusals.

A useful assistant needs:

text
Safe when necessary
+
Helpful when allowed

37. Over-Refusal

A model may become too cautious.

Example:

text
User:
Explain how encryption works.

Model:
I cannot help with cybersecurity.

This is an over-refusal.

A better model distinguishes:

text
benign educational request
vs.
harmful operational request

This requires high-quality policy data and nuanced evaluation.

38. Under-Refusal

The opposite failure:

Architecture & Data Flow
Unsafe request
 |
 v
Model provides dangerous assistance

Alignment therefore requires both:

text
helpfulness
and
appropriate safety boundaries

39. Alignment Tax

Alignment can sometimes reduce performance on capabilities that were strong in the base model.

This is sometimes called an alignment tax.

Conceptually:

Architecture & Data Flow
Base model
 |
 +--> broad capability
 |
 v
Aggressive alignment
 |
 +--> better safety
 +--> better instruction following
 |
 +--> possible capability regression

This trade-off must be measured rather than assumed.

40. Capability Preservation

After post-training, compare:

text
Base model
vs.
Post-trained model

Evaluate:

knowledge
coding
mathematics
reasoning
multilingual performance
long context
tool use
safety
instruction following

The goal is not merely higher alignment scores.

41. Alignment Evaluation Matrix

A useful matrix:

Dimension	Example Metric
Helpfulness	Human preference
Correctness	Expert evaluation
Safety	Policy benchmark
Instruction following	Task success
Factuality	Grounded evaluation
Coding	Unit-test pass rate
Structured output	Schema validity
Tool use	Successful tool trajectory
Refusal quality	Safety + helpfulness
Robustness	Adversarial evaluation

42. Human Evaluation

Human evaluation remains valuable for qualities that are difficult to fully automate.

Annotators can rate:

text
Correctness
Relevance
Clarity
Helpfulness
Safety
Style

Use clear rubrics.

For example:

Mathematical Formulation
5 = excellent
4 = good
3 = acceptable
2 = weak
1 = unacceptable

Inter-annotator agreement should be monitored.

43. LLM-as-a-Judge

An LLM can score responses using a rubric.

Example:

text
Evaluate:
1. factual correctness
2. relevance
3. clarity
4. safety

Return JSON.

Useful for scale, but judge behavior must be calibrated against human judgments.

Potential biases include:

verbosity preference
position bias
style preference
self-preference
prompt sensitivity

44. Preference Data Quality

Bad preference labels can produce bad alignment.

Potential problems:

text
Annotator disagreement
Ambiguous instructions
Poor rubric
Judge bias
Position bias
Low-quality candidates

Improve the pipeline with:

clear rubrics
annotator training
multiple labels
adjudication
agreement analysis
difficult-example sampling

45. Pairwise Evaluation

Instead of asking:

›Is response A a 4/5?

ask:

text
Which response is better?

A
or
B

Pairwise evaluation can be easier for humans and useful for preference learning.

46. Tool-Use Post-Training

An assistant may need to learn:

text
When to call a tool
What arguments to provide
How to interpret results
When not to call a tool
How to answer after tool execution

Example:

Architecture & Data Flow
User:
What is the weather tomorrow?

Model
 |
 v
Weather tool
 |
 v
Tool result
 |
 v
Final response

Training data can contain complete tool-use trajectories.

47. Tool-Calling Dataset

A training record might represent:

json
{
 "messages": [
 {
 "role": "user",
 "content": "Find my order."
 },
 {
 "role": "assistant",
 "tool_call": {
 "name": "get_order",
 "arguments": {
 "order_id": "..."
 }
 }
 },
 {
 "role": "tool",
 "content": "Order delivered."
 },
 {
 "role": "assistant",
 "content": "Your order was delivered."
 }
 ]
}

Sensitive identifiers should be replaced with safe placeholders in training examples.

48. Structured-Output Training

Models can be post-trained to produce schemas such as:

json
{
 "topic": "linear_regression",
 "difficulty": "beginner",
 "question_type": "conceptual"
}

Training should include:

valid examples
invalid examples
edge cases
optional fields
nested structures

Evaluation should validate actual schema compliance.

49. Function Calling

A model may learn a tool specification:

json
{
 "name": "search_documents",
 "parameters": {
 "query": "string"
 }
}

Post-training can teach:

Architecture & Data Flow
natural language
 |
 v
correct function
 |
 v
valid arguments

But runtime validation is still required.

The model's learned behavior is not a security boundary.

50. Tool Safety

A post-trained model should not have unlimited authority.

Use:

Architecture & Data Flow
Model
 |
 v
Policy layer
 |
 v
Argument validator
 |
 v
Permission check
 |
 v
Tool

This separates model behavior from actual authorization.

51. Educational AI Alignment

Educational assistants need specialized post-training objectives.

Examples:

text
Explain without unnecessary complexity
Ask guiding questions
Give hints before solutions
Adapt difficulty
Identify misconceptions
Encourage learning
Avoid doing graded work dishonestly

This is different from optimizing generic helpfulness.

52. Educational Preference Data

Consider:

›Prompt:
I don't understand gradient descent.

Response A:

›Gradient descent is an optimization algorithm...

Response B:

›Think of yourself walking down a hill...

A learner may prefer B depending on level.

The preference dataset should include:

learner level
learning objective
pedagogical strategy
correctness

53. Socratic Training

An educational model may be trained to ask guiding questions.

Example:

Mathematical Formulation
Student:
What is the answer to 2x + 5 = 15?

Assistant:
What would you do first to remove the +5?

This behavior can be learned through SFT and preference data.

54. Hint vs Answer Policy

A learning assistant can be trained on levels:

Architecture & Data Flow
Hint 1
 |
 v
Hint 2
 |
 v
Detailed explanation
 |
 v
Full solution

The model learns to avoid immediately giving the final answer when pedagogically inappropriate.

55. Post-Training Data Pipeline

A production pipeline can look like:

Architecture & Data Flow
Base Model
 |
 v
Task Specification
 |
 v
SFT Data
 |
 v
Supervised Fine-Tuning
 |
 v
Candidate Generation
 |
 v
Preference Collection
 |
 +--> Human
 +--> Expert
 +--> Judge
 |
 v
Preference Training
 |
 v
Safety Training
 |
 v
Tool Training
 |
 v
Evaluation
 |
 v
Red Teaming
 |
 v
Release Candidate

56. Version Everything

Post-training experiments should version:

text
base model
SFT dataset
preference dataset
chat template
tokenizer
training code
hyperparameters
evaluation suite
safety policy
tool schemas

A model without lineage is difficult to reproduce.

57. Post-Training Experiment Tracking

Record:


🐍 PythonInteractive WebAssembly
run = {
 "base_model": "base-v1",
 "sft_dataset": "sft-v4",
 "preference_dataset": "prefs-v2",
 "method": "DPO",
 "learning_rate": 5e-6,
 "epochs": 2,
 "evaluation": "alignment-suite-v3",
}

Store the configuration with the resulting checkpoint.

58. Hyperparameters

Important SFT parameters include:

learning rate
batch size
epochs
sequence length
warmup
weight decay
gradient accumulation
maximum gradient norm

Preference methods may introduce additional parameters such as:

preference temperature
reference-model weighting
KL coefficient
reward scaling
rollout parameters

The correct values are empirical.

59. Catastrophic Forgetting

Aggressive post-training can reduce some capabilities learned during pretraining.

Example:

Mathematical Formulation
Before:
coding = strong
general knowledge = strong

After narrow SFT:
coding = strong
general knowledge = weaker

Mitigations include:

diverse data
lower learning rates
mixed-domain examples
regularization
reference-model constraints
capability regression tests

60. Distribution Shift

Training data may not represent production usage.

Example:

text
Training:
short questions

Production:
long documents + ambiguous requests

The model may perform poorly after deployment.

Post-training data should reflect real workload distributions.

61. Online Feedback

Production systems can collect feedback:

Architecture & Data Flow
User interaction
 |
 v
Feedback
 |
 v
Error taxonomy
 |
 v
Candidate training examples
 |
 v
Post-training
 |
 v
New model

Privacy and governance controls are essential.

62. Continual Post-Training

Models can be updated periodically.

Architecture & Data Flow
Model v1
 |
 v
Production feedback
 |
 v
Dataset v2
 |
 v
Post-training
 |
 v
Model v2

Do not automatically train on raw user conversations.

First apply:

privacy filtering
consent and policy checks
quality filtering
deduplication
annotation
evaluation

63. Alignment Regression

A new model may improve one metric while damaging another.

Example:

text
Helpfulness: +5%
Safety: +3%
Coding: -8%
Long context: -6%

Therefore every release should run a regression suite.

64. Safety Regression

Test:

text
Known unsafe prompts
Known benign prompts
Adversarial prompts
Prompt injection
Sensitive-data requests
Tool abuse

The goal is to detect:

›new unsafe behavior

as well as:

›new over-refusal

65. Alignment Under Adversarial Pressure

A model may behave correctly on normal prompts but fail under:

text
role-play
prompt injection
multi-turn manipulation
conflicting instructions
encoded text
context poisoning

Post-training should include adversarial evaluation.

66. Jailbreak Resistance

A jailbreak attempts to bypass model policies.

A robust post-training program should not rely on one refusal phrase.

It should teach the model to:

text
recognize unsafe intent
+
follow higher-priority policy
+
avoid revealing protected information
+
provide safe alternatives where appropriate

Application-level controls remain necessary.

67. Alignment and System Prompts

Post-training is not a replacement for system-level controls.

A production system can use:

text
Model behavior
 +
System prompt
 +
Policy engine
 +
Tool permissions
 +
Input/output filters
 +
Monitoring

Defense in depth is stronger than relying on model behavior alone.

68. Alignment for Different Personas

A model may need different behaviors for:

text
Student
Teacher
Administrator
Developer
Customer
Researcher

Some differences should be controlled through prompts and application policy rather than creating separate models.

Use post-training when behavior should be intrinsic and stable.

69. When to Use SFT

SFT is particularly useful when:

›desired behavior is demonstrable

Examples:

formatting
tool calls
domain responses
style
instruction following

If you can write a good target response, SFT is often a natural starting point.

70. When to Use Preference Optimization

Preference methods are useful when:

›multiple answers are possible

and the challenge is selecting the better one.

Examples:

helpfulness
style
safety
concise vs verbose
pedagogical quality

71. When to Use RL-Style Optimization

RL-style methods can be useful when:

the reward can be evaluated
the task has sequential behavior
online optimization is valuable
preference optimization alone is insufficient

Examples may include:

tool-use trajectories
complex agent behavior
environments with executable rewards

But RL adds substantial complexity.

72. Reward Model vs Direct Preference

A useful conceptual comparison:

Architecture & Data Flow
Reward-model pipeline:

Preference
 |
 v
Reward model
 |
 v
RL optimization
 |
 v
Model

versus:

Architecture & Data Flow
DPO-style:

Preference
 |
 v
Direct preference objective
 |
 v
Model

The second pipeline can be simpler, while the first provides an explicit reward model that can be reused for scoring.

73. Post-Training Cost

Cost depends on:

text
base model size
dataset size
sequence length
number of candidates
teacher inference
human annotation
GPU hours
evaluation

Preference data can be particularly expensive because each prompt may require multiple candidate generations.

74. Efficient Candidate Generation

Instead of generating 20 candidates with an expensive model:

›Large teacher -> 20 candidates

use:

Architecture & Data Flow
Smaller generator -> 10 candidates
 |
 v
Large judge -> select

or:

Architecture & Data Flow
Small model
 |
 v
candidate filtering
 |
 v
large model refinement

This reduces cost.

75. Human-in-the-Loop Alignment

A mature pipeline can route uncertain examples to humans.

Architecture & Data Flow
Generated pair
 |
 v
Automated judge
 |
 +--> high confidence -> accept
 |
 +--> uncertain -> human
 |
 v
 label

This concentrates human effort where it adds the most value.

76. Active Preference Learning

Instead of labeling random examples, select examples where:

judges disagree
models disagree
reward margin is small
safety confidence is low
new capabilities are emerging

Architecture & Data Flow
Candidate pool
 |
 v
Uncertainty scoring
 |
 v
Human annotation
 |
 v
Preference dataset

This can improve annotation efficiency.

77. Preference Margin

Suppose a reward model estimates:

Mathematical Formulation
A = 0.81
B = 0.79

The difference is small.

This may be an uncertain example.

If:

Mathematical Formulation
A = 0.95
B = 0.10

the preference is much clearer.

Margin-based sampling can prioritize ambiguous cases for human review.

78. Post-Training Data Balance

A dataset should balance:

text
helpfulness
correctness
safety
diversity
formatting
edge cases

Overemphasizing one objective can produce undesirable behavior.

For example:

Architecture & Data Flow
Too much safety data
 -> over-refusal

Too much verbosity
 -> bloated answers

Too much narrow domain data
 -> capability forgetting

79. Alignment Tax Measurement

Compare:

›Base model
Post-trained model

using the same capability suite.

Track:

text
Capability gain
Safety gain
Instruction gain
Regression

A useful release decision asks:

Did the behavioral improvement justify the capability trade-off?

80. End-to-End Educational AI Post-Training

Architecture & Data Flow
Base LLM
 |
 v
Educational SFT
 |
 +--> explanations
 +--> hints
 +--> quizzes
 +--> misconceptions
 |
 v
Preference Data
 |
 +--> pedagogical quality
 +--> correctness
 +--> level appropriateness
 |
 v
Preference Optimization
 |
 v
Safety Alignment
 |
 v
Tool Training
 |
 +--> curriculum lookup
 +--> progress lookup
 +--> quiz generation
 |
 v
Educational Evaluation
 |
 v
Production Tutor

81. Educational Evaluation

Evaluate:

text
Concept correctness
Pedagogical quality
Difficulty alignment
Hint quality
Misconception handling
Student engagement
Academic integrity
Safety

A generic chatbot benchmark is not enough.

82. Example Educational Preference

Prompt:

›I still don't understand why overfitting is bad.

Response A:

›Overfitting occurs when...

Response B:

›Imagine practicing one exact exam paper until you memorize every answer...

If the learner is a beginner, B may be preferred because it uses an intuitive analogy.

The preference dataset should encode the target educational behavior.

83. Production Release Pipeline

A model release might follow:

Architecture & Data Flow
Candidate Model
 |
 v
Unit Tests
 |
 v
Offline Evaluation
 |
 v
Safety Evaluation
 |
 v
Red Teaming
 |
 v
Regression Testing
 |
 v
Canary Deployment
 |
 v
Online Monitoring
 |
 v
Full Release

A model should not move directly from training to unrestricted production.

84. Rollback

Every post-trained model should have a known previous version.

text
v1.2 <-- production
v1.3 <-- candidate
v1.4 <-- training

If v1.3 introduces regressions:

›rollback -> v1.2

This requires versioned model artifacts and deployment configuration.

85. Post-Training Governance

Track:

text
Data provenance
Model provenance
Annotation policy
Safety policy
Evaluation results
Known limitations
Release decision
Approvals

This becomes especially important for enterprise and educational deployments.

86. Practical Project 1: Instruction Tuning

Create a small SFT dataset.

Include:

text
explanation
summarization
classification
structured output
coding

Train a small model.

Compare:

text
base model
vs.
SFT model

Evaluate instruction following.

87. Practical Project 2: Preference Dataset

For each prompt:

generate 3 responses
score them
select the preferred response
create chosen/rejected pairs
train a preference-optimized model
compare against SFT

Measure:

human preference
correctness
verbosity
formatting

88. Practical Project 3: DPO Experiment

Build:

Architecture & Data Flow
SFT model
 |
 v
Preference dataset
 |
 v
DPO
 |
 v
DPO model

Compare:

text
SFT
vs.
DPO

using the same evaluation set.

89. Practical Project 4: Rejection Sampling

Generate:

›10 responses / prompt

Score each response.

Keep the best.

Create an SFT dataset.

Compare:

text
original SFT
vs.
rejection-sampled SFT

90. Practical Project 5: Educational Alignment

Create a dataset for:

text
hints
explanations
misconception correction
difficulty adaptation

Train a small assistant.

Evaluate with teacher-created rubrics.

91. Practical Project 6: Tool-Use Post-Training

Create examples for:

text
search
database lookup
calculator
curriculum lookup

Train the model to:

text
choose tool
produce valid arguments
interpret results
respond correctly

Add runtime validation.

92. Advanced Exercise: Alignment Tax

Evaluate a model before and after post-training.

Measure:

text
helpfulness
safety
coding
math
general knowledge
long context

Identify:

›improvements
regressions

Discuss whether the trade-off is acceptable.

93. Advanced Exercise: Reward Hacking

Construct a toy reward model that favors verbosity.

Train or optimize a small model against it.

Observe whether outputs become:

›longer
but not better

Then redesign the reward signal.

This demonstrates proxy optimization.

94. Advanced Exercise: Human vs LLM Preferences

Create a small evaluation set.

Collect:

›human preference
LLM-judge preference

Measure agreement.

Analyze disagreements.

Questions:

Does the judge prefer verbosity?
Does it favor a particular style?
Does it miss factual errors?
Does it behave differently on safety examples?

95. Advanced Exercise: Safety / Helpfulness Frontier

Create candidate models with different alignment strengths.

Measure:

text
helpfulness
safety
over-refusal
under-refusal

Plot the trade-off.

The objective is not:

›maximum refusal

but:

›appropriate behavior

96. Advanced Exercise: Iterative Post-Training

Build:

Architecture & Data Flow
Model v1
 |
 v
Evaluation
 |
 v
Failure analysis
 |
 v
Targeted dataset
 |
 v
Post-training
 |
 v
Model v2

Repeat for three iterations.

Track whether each iteration improves the target capability without causing regressions.

97. Common Mistakes

Mistake 1: Treating SFT as the entire alignment problem#

SFT teaches demonstrations but may not capture nuanced preferences.

Mistake 2: Using low-quality preference data#

Preference optimization can amplify annotation errors.

Mistake 3: Trusting a reward model blindly#

Reward models are proxies and can be exploited.

Mistake 4: Optimizing one metric#

A model can become safer while becoming less useful.

Mistake 5: Overtraining on narrow data#

This can cause capability regression.

Mistake 6: Ignoring over-refusal#

Safety alignment should distinguish harmful requests from legitimate ones.

Mistake 7: Training tool use without runtime controls#

Model behavior is not an authorization mechanism.

Mistake 8: Using raw production conversations#

Privacy, consent, security, and quality filtering must be addressed first.

Mistake 9: Evaluating only after training#

Run evaluations throughout the post-training lifecycle.

Mistake 10: Forgetting the base model#

Post-training should be evaluated against the original model to understand capability changes.

98. Practical Post-Training Checklist

Before SFT:

text
[ ] Base model identified
[ ] Tokenizer verified
[ ] Chat template verified
[ ] SFT schema validated
[ ] Data quality checked
[ ] Evaluation set isolated
[ ] Safety examples reviewed

Before preference training:

text
[ ] Preference rubric defined
[ ] Candidate generation tested
[ ] Human/LLM agreement measured
[ ] Chosen/rejected format validated
[ ] Bias checks performed

Before release:

text
[ ] Capability evaluation
[ ] Safety evaluation
[ ] Regression testing
[ ] Tool-use evaluation
[ ] Adversarial evaluation
[ ] Human evaluation
[ ] Canary plan
[ ] Rollback plan

99. Final Mental Model

Think of post-training as shaping a powerful but general model into a reliable product behavior.

Architecture & Data Flow
 BASE MODEL
 broad capabilities
 |
 v
 +----------------------+
 | Supervised Fine- |
 | Tuning |
 +----------+-----------+
 |
 v
 INSTRUCTION MODEL
 |
 v
 +----------------------+
 | Preference Learning |
 | SFT / DPO / RLHF |
 +----------+-----------+
 |
 v
 ALIGNED MODEL
 |
 +----------+-----------+
 | | |
 v v v
 Safety Tools Structured
 behavior behavior outputs
 | | |
 +----------+------------+
 |
 v
 EVALUATION
 |
 v
 PRODUCTION MODEL

The most important distinction is:

Pretraining teaches the model to model language; post-training teaches it how to behave for a particular purpose.

A successful post-training system balances:

text
Capability
+
Helpfulness
+
Correctness
+
Safety
+
Instruction following
+
Tool reliability
+
Generalization

Key Takeaways

Pretraining and post-training solve different problems.
SFT teaches models from high-quality demonstrations.
Chat templates and response masking are important implementation details.
Preference data teaches relative quality rather than only absolute target responses.
Reward models are useful but represent proxy objectives.
RLHF combines preference learning with policy optimization.
PPO-style methods constrain policy updates to reduce unstable behavior.
DPO directly optimizes preference pairs without requiring the same reward-model-plus-online-RL pipeline.
Rejection sampling can create strong training data from multiple candidate responses.
Safety alignment should minimize both harmful assistance and unnecessary refusal.
Post-training can cause an alignment tax, so capability regression must be measured.
Tool-use training teaches when and how to call tools, but runtime authorization must remain outside the model.
Educational AI requires specialized alignment objectives such as hints, misconception handling, and difficulty adaptation.
Human evaluation remains important for nuanced preferences.
LLM-as-a-Judge can scale evaluation but must be calibrated.
Reward hacking demonstrates why proxy objectives require careful design.
Continual post-training should use governed, filtered, and versioned production feedback.
Every post-training release should pass capability, safety, adversarial, and regression evaluations.
Model lineage should include base model, datasets, tokenizer, training configuration, and evaluation suite.
The goal of alignment is not maximum obedience or maximum refusal; it is reliable behavior under the intended policy and workload.

Knowledge Check

Question 1#

Why is pretraining alone insufficient for building a production assistant?

Question 5#

What does preference data add beyond ordinary SFT?

Question 12#

Why is model behavior not sufficient for tool security?

Course Progression

Completed:

text
01 Generative AI & LLM Foundations
02 Transformers & LLM Architecture
03 RAG, Embeddings & Vector Databases
04 LangChain, LangGraph & Agentic AI
05 LLM Evaluation, Safety & Guardrails
06 Multimodal Generative AI
07 Fine-Tuning, LoRA, QLoRA & PEFT
08 Open-Source, Open-Weight & Sovereign LLMs
09 LLMOps & Inference Optimization
10 End-to-End Generative AI Projects
11 AI Application Security & Governance
12 Advanced RAG & Agent Architectures
13 AI Platform Architecture & Engineering
14 Distributed Inference & GPU Engineering
15 Data Engineering & Evaluation Infrastructure
16 Advanced Evaluation & Benchmarking
17 Synthetic Data & Dataset Generation
18 Knowledge Distillation & Model Compression
19 Advanced LLM Training
20 Post-Training & Alignment

›21 LLM Reasoning & Reasoning Models

The next notebook moves into reasoning-focused language models: chain-of-thought concepts, test-time compute, reasoning traces, verifier models, self-consistency, search-based reasoning, process supervision, outcome supervision, reasoning data, reward signals, inference-time scaling, and evaluation of reasoning systems.

Knowledge Checkpoint

Post-Training, RLHF & DPO Checkpoint

Q1.How does Direct Preference Optimization (DPO) simplify standard RLHF (Reinforcement Learning from Human Feedback)?

ADPO derives an exact closed-form mapping between the reward function and optimal policy, optimizing preference loss directly on paired data $(y_w, y_l)$ without training a separate reward model or using unstable PPO reinforcement learning.

BDPO removes the need for human preference data.

CDPO fine-tunes only the tokenizer.

DDPO trains exclusively with unsupervised clustering.

Q2.What is the purpose of the KL-divergence penalty in RLHF / PPO alignment?

ATo prevent the policy model from drifting too far from the original supervised fine-tuned (SFT) base model (preventing reward hacking and mode collapse).

BTo speed up GPU inference latency.

CTo enforce grammatical punctuation.

DTo increase token vocabulary size.

Q3.What is Supervised Fine-Tuning (SFT) in the standard LLM post-training pipeline?

ATraining the pretrained base model on high-quality curated instruction-response demonstration dialogues using cross-entropy loss.

BEvaluating models on benchmark leaderboards.

CQuantizing weights to 4-bit integers.

DGenerating synthetic embeddings.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

Advanced LLM Training

Next Lesson

Knowledge Distillation & Model Compression for Generative AI