Project 5: Text Generation with RLHF

Duration: 3 weeks Points: 100 Prerequisites: Complete Lessons 6 (PPO), 15 (LLMs), 16 (RLHF) Difficulty: Advanced

Project Overview

In this project, you'll implement the complete 3-stage RLHF pipeline to align a language model with human preferences. You'll perform supervised fine-tuning (SFT), train a reward model from preference comparisons, and use PPO to optimize the model for helpfulness, harmlessness, and honesty. This is the exact technique used to create ChatGPT, Claude, and other aligned AI assistants.

Why This Matters: RLHF is THE breakthrough that transformed language models from text predictors into helpful assistants. This project gives you production-ready experience with the most important technique in modern AI.

What You'll Build:

• Complete RLHF pipeline (SFT -> Reward Model -> PPO)
• Aligned language model for instruction following
• Evaluation suite comparing base, SFT, and RLHF models
• Reward hacking detection and mitigation system
• Portfolio-ready demo showcasing alignment improvements

Learning Objectives

By completing this project, you will:

1. Implement the 3-stage RLHF pipeline from scratch
2. Train reward model on human preference comparisons (Bradley-Terry model)
3. Apply PPO with KL divergence penalty to language models
4. Evaluate model alignment using helpfulness, harmlessness, honesty metrics
5. Debug reward hacking behaviors and apply mitigation strategies
6. Deploy aligned model as interactive chatbot

Requirements

Functional Requirements

Your RLHF system must:

• Stage 1 (SFT): Fine-tune base model on instruction dataset (10K+ examples)
• Stage 2 (Reward Model): Train on preference pairs ``with >65``% validation accuracy
• Stage 3 (PPO): Optimize policy with KL penalty (β = 0.01-0.1)
• Demonstrate improvement: RLHF model preferred over SFT ``model >=75``% of time (human or automated eval)
• Prevent reward hacking: KL divergence stays <0.5 during training
• Interactive demo: Chatbot interface comparing base vs SFT vs RLHF responses

Technical Requirements

Your implementation must include:

• SFT: HuggingFace Trainer for instruction tuning
• Reward Model: Bradley-Terry preference learning
• PPO: TRL library or custom implementation with:
  • Clipped surrogate objective
  • KL divergence penalty
  • Entropy bonus for exploration
  • Value function baseline
• Evaluation: Win rate, perplexity, reward distributions
• Logging: Track policy loss, value loss, KL, entropy, rewards

Code Structure

project-05-text-generation-rlhf/
├── README.md                        # Project documentation
├── requirements.txt                 # Python dependencies
├── stage1_sft.py                   # Supervised fine-tuning
├── stage2_reward_model.py          # Reward model training
├── stage3_ppo.py                   # PPO training loop
├── evaluation.py                   # Model comparison and metrics
├── chatbot_demo.py                 # Interactive Gradio interface
├── data/                           # Datasets
│   ├── instructions/               # SFT data
│   └── preferences/                # Reward model data
├── models/                         # Saved checkpoints
│   ├── sft_model/
│   ├── reward_model/
│   └── rlhf_model/
└── logs/                           # Training logs

Grading Rubric

Bonus Points (+10 each):

• Implement Constitutional AI (self-critique loop)
• Implement DPO (Direct Preference Optimization) as alternative to PPO
• Train on larger model (7B+ parameters with QLoRA)
• Deploy to public demo (Hugging Face Spaces)

Milestones

Week One: Supervised Fine-Tuning (SFT)

Day 1-2: Dataset Preparation

• Download Alpaca, Dolly, or OpenAssistant datasets
• Format data: instruction, response pairs
• Split into train/val/test

Day 3-4: SFT Training

• Load base model (GPT-2, Llama-2-7B, or similar)
• Fine-tune on instruction dataset
• Monitor loss and sample generations

Day 5: SFT Evaluation

• Test on held-out instructions
• Compare to base model
• Save SFT checkpoint

Deliverable: SFT model that follows basic instructions

Week 2: Reward Model Training

Day 6-7: Preference Dataset

• Download Anthropic HH-RLHF or OpenAssistant preferences
• Verify data format: prompt, chosen, rejected
• Balance dataset

Day 8-9: Reward Model Implementation

• Implement Bradley-Terry model (reward head on SFT backbone)
• Train on preference comparisons
• Monitor validation accuracy

Day 10: Reward Model Evaluation

• Test on held-out comparisons
• Analyze reward distributions (chosen > rejected?)
• Save reward model checkpoint

Deliverable: Reward model ``with >65``% validation accuracy

Week 3: PPO Fine-Tuning and Evaluation

Day 11-13: PPO Implementation

• Set up TRL PPOTrainer or implement custom loop
• Configure hyperparameters (β, learning rate, batch size)
• Start PPO training with reward model

Day 14-15: Training Monitoring

• Watch for reward hacking (sudden reward spikes)
• Monitor KL divergence (should stay <0.5)
• Tune β if needed

Day 16-17: Evaluation

• Generate responses from base, SFT, RLHF models
• Human or automated preference evaluation
• Calculate win rates and metrics

Day 18-19: Interactive Demo

• Build Gradio chatbot interface
• Side-by-side comparison of models
• Deploy locally or to cloud

Day 20-21: Documentation and Portfolio

• Write comprehensive report
• Record demo video
• Prepare presentation

Deliverable: Complete RLHF system with evaluation and demo

Implementation Guide

Stage One: Supervised Fine-Tuning

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments

# Load base model
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# Format dataset
def format_instruction(example):
    return {
        "text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    }

dataset = load_dataset("tatsu-lab/alpaca").map(format_instruction)

# Training arguments
training_args = TrainingArguments(
    output_dir="./sft_model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-5,
    logging_steps=100,
    save_steps=1000,
)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
)
trainer.train()

Stage 2: Reward Model Training

import torch.nn as nn

class RewardModel(nn.Module):
    def __init__(self, base_model):
        super().__init__()
        self.backbone = base_model
        self.reward_head = nn.Linear(base_model.config.hidden_size, 1)

    def forward(self, input_ids, attention_mask):
        outputs = self.backbone(input_ids, attention_mask=attention_mask)
        # Use last token's hidden state
        last_hidden = outputs.last_hidden_state[:, -1, :]
        reward = self.reward_head(last_hidden)
        return reward

# Training loop
for chosen, rejected in preference_dataloader:
    # Compute rewards
    r_chosen = reward_model(chosen['input_ids'], chosen['attention_mask'])
    r_rejected = reward_model(rejected['input_ids'], rejected['attention_mask'])

    # Bradley-Terry loss
    loss = -torch.log(torch.sigmoid(r_chosen - r_rejected)).mean()

    # Backward
    loss.backward()
    optimizer.step()

Stage 3: PPO Training

from trl import PPOTrainer, PPOConfig

# Load models
policy_model = AutoModelForCausalLMWithValueHead.from_pretrained("./sft_model")
ref_model = AutoModelForCausalLM.from_pretrained("./sft_model")  # Frozen reference
reward_model = RewardModel.from_pretrained("./reward_model")

# PPO config
ppo_config = PPOConfig(
    learning_rate=1e-5,
    batch_size=16,
    mini_batch_size=4,
    init_kl_coef=0.05,  # KL penalty coefficient
    ppo_epochs=4,
)

# Initialize trainer
ppo_trainer = PPOTrainer(
    config=ppo_config,
    model=policy_model,
    ref_model=ref_model,
    tokenizer=tokenizer,
)

# Training loop
for prompts in prompt_dataloader:
    # Generate responses from policy
    query_tensors = [tokenizer.encode(p, return_tensors="pt")[0] for p in prompts]
    response_tensors = ppo_trainer.generate(query_tensors, max_new_tokens=128)

    # Compute rewards from reward model
    rewards = []
    for query, response in zip(query_tensors, response_tensors):
        full_text = tokenizer.decode(torch.cat([query, response]))
        reward = reward_model(tokenizer(full_text, return_tensors="pt"))
        rewards.append(reward)

    # PPO update
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)

    # Log
    print(f"Mean reward: {stats['ppo/mean_scores']:.2f}, KL: {stats['ppo/mean_non_score_reward']:.4f}")

Hyperparameter Recommendations

Evaluation Metrics

Win Rate (Gold Standard)

def evaluate_win_rate(model_a, model_b, test_prompts, num_samples=100):
    """
    Present responses from both models to human evaluator
    Calculate percentage where model_a is preferred
    """
    wins = 0
    for prompt in test_prompts[:num_samples]:
        response_a = generate(model_a, prompt)
        response_b = generate(model_b, prompt)

        # Human evaluation (or automated with GPT-4)
        preference = get_preference(prompt, response_a, response_b)

        if preference == "A":
            wins += 1

    return wins / num_samples

# Expected: RLHF > SFT with ≥75% win rate

Automated Evaluation (GPT-4 as Judge)

def gpt4_judge(prompt, response_a, response_b):
    """Use GPT-4 to evaluate which response is better"""
    judgment_prompt = f"""
    Evaluate which response is more helpful, harmless, and honest.

    Prompt: {prompt}

    Response A: {response_a}
    Response B: {response_b}

    Which response is better? (A/B/Tie)
    """
    judgment = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": judgment_prompt}]
    )
    return judgment.choices[0].message.content.strip()

Reward Hacking Detection

Signs of Reward Hacking

• Sudden reward spikes during PPO training
• Responses become longer without adding value
• Responses contain repetitive keywords ("helpful", "safe", "accurate")
• KL divergence explodes (>1.0)

Mitigation Strategies

1. Increase β: Higher KL penalty prevents drift
2. Length penalty: reward = reward - 0.01 * length
3. Diversity penalty: Penalize repetition
4. Adversarial training: Add hacked examples to reward model training

Resources

Documentation

• HuggingFace TRL - RLHF training library
• HuggingFace Transformers - Language models

Research Papers

• InstructGPT (Ouyang et al., 2022): Training LLMs to follow instructions
• Constitutional AI (Bai et al., 2022): Harmlessness from AI Feedback
• DPO (Rafailov et al., 2023): Direct Preference Optimization

Datasets

• Anthropic HH-RLHF
• Alpaca
• OpenAssistant

Submission Guidelines

Required Deliverables:

1. Code repository with all 3 stages
2. Trained models (SFT, reward model, RLHF final)
3. Evaluation report with win rates and metrics
4. Interactive chatbot demo (Gradio)
5. Analysis of reward hacking (if observed)

Deadline: 3 weeks from project start

Portfolio Presentation

Demo Website:

• Title: "RLHF Chatbot - Aligned Language Model"
• Side-by-side comparison interface
• Gallery showing improvement examples
• Technical highlights: "3-stage RLHF with 78% win rate over SFT"

LinkedIn/Resume:

"Implemented complete RLHF pipeline to align language model with human preferences, achieving 78% win rate improvement over supervised fine-tuning baseline. Deployed interactive chatbot demonstrating helpfulness, harmlessness, and honesty alignment."

Good luck! RLHF is the most important technique in modern AI. This project will give you the skills to build aligned AI systems.

Related Projects:

• Project 2 - Autonomous Robot Navigation <- (PPO for robotics)
• Project 6 - Multi-Modal Content Generator -> (RLHF for vision + language)