Practice and reinforce the concepts from Lesson 9
In this activity, you'll build a working generative model system that demonstrates the core concepts from Lesson 9. You'll implement a Gaussian Mixture Model (GMM) from scratch, visualize latent spaces, and create an interactive tool to explore generative vs discriminative models.
By completing this activity, you will:
Download the activity template from the Templates folder:
AI25-Template-activity-09-introduction-to-generative-models.zipTemplates/AI25-Template-activity-09-introduction-to-generative-models.zipactivity-09-introduction-to-generative-models.ipynb to Google ColabExecute the first few cells to:
Scenario: Implement GMM to model 2D data with multiple clusters.
TODO 1: Implement E-step (compute responsibilities)
class GMM:
def _e_step(self, X):
"""
E-step: Compute responsibilities
Args:
X: Data (n_samples, n_features)
Returns:
responsibilities: (n_samples, n_components)
"""
# TODO 1: Compute probability of each point under each component
# Hint: Use scipy.stats.multivariate_normal.pdf
# responsibilities[i, k] = P(component k | point i)
responsibilities = np.zeros((X.shape[0], self.n_components))
# Your code here
return responsibilities
TODO 2: Implement M-step (update parameters)
def _m_step(self, X, responsibilities):
"""
M-step: Update parameters
Args:
X: Data (n_samples, n_features)
responsibilities: (n_samples, n_components)
"""
# TODO 2: Update parameters based on responsibilities
# weights: π_k = (sum of responsibilities for k) / n_samples
# means: μ_k = weighted average of points (weights = responsibilities)
# covariances: Σ_k = weighted covariance matrix
# Your code here
pass
TODO 3: Generate new samples from trained GMM
def sample(self, n_samples):
"""
Generate samples from the GMM
Args:
n_samples: Number of samples to generate
Returns:
samples: (n_samples, n_features)
"""
samples = []
for _ in range(n_samples):
# TODO 3: Implement sampling
# Step 1: Choose component k with probability self.weights[k]
# Step 2: Sample from N(self.means[k], self.covariances[k])
# Your code here
pass
return np.array(samples)
Interactive visualization showing:
Features:
TODO 4: Implement both approaches for 2-class classification
Discriminative (Logistic Regression):
def discriminative_model(X_train, y_train, X_test):
"""
Train logistic regression (discriminative)
Learns: P(y | X)
"""
# TODO 4a: Fit logistic regression
# Use sklearn.linear_model.LogisticRegression
# Your code here
pass
Generative (Gaussian Naive Bayes):
def generative_model(X_train, y_train, X_test):
"""
Train Gaussian Naive Bayes (generative)
Learns: P(X | y) and P(y), then computes P(y | X) via Bayes' rule
"""
# TODO 4b: Fit Gaussian Naive Bayes
# Use sklearn.naive_bayes.GaussianNB
# Your code here
pass
TODO 5: Build interactive latent space explorer for VAE-style model
class LatentSpaceExplorer:
def __init__(self, model):
self.model = model
def interpolate(self, z1, z2, num_steps=10):
"""
Interpolate between two latent codes
Args:
z1, z2: Start and end latent codes
num_steps: Number of interpolation steps
Returns:
interpolated_samples: Generated samples along path
"""
# TODO 5a: Implement linear interpolation in latent space
# z_interpolated = (1-alpha) * z1 + alpha * z2
# Then decode each z_interpolated
# Your code here
pass
def arithmetic(self, z_base, z_attribute, alpha=1.0):
"""
Latent space arithmetic
Args:
z_base: Base latent code
z_attribute: Attribute to add/subtract
alpha: Strength of attribute
Returns:
generated_sample: Result of z_base + alpha * z_attribute
"""
# TODO 5b: Implement latent arithmetic
# z_new = z_base + alpha * z_attribute
# Decode z_new
# Your code here
pass
Pre-built functions for:
After training on 2D data with 3 clusters:
Iteration 1: Log-likelihood = -2.45
Iteration 10: Log-likelihood = -1.82
Iteration 20: Log-likelihood = -1.75 (converged)
✓ Learned 3 components
✓ Component 1: μ=[2.1, 3.5], weight=0.33
✓ Component 2: μ=[5.8, 1.2], weight=0.34
✓ Component 3: μ=[1.5, 8.7], weight=0.33
Generated 100 samples:
✓ Sample distribution matches training data
✓ All 3 modes represented
✓ Visual quality: realistic clusters
Interactive plot showing:
2-class classification results:
Discriminative (Logistic Regression):
- Train accuracy: 92%
- Test accuracy: 89%
- Decision boundary: linear
Generative (Gaussian Naive Bayes):
- Train accuracy: 88%
- Test accuracy: 86%
- Decision boundary: curved
- Bonus: Can generate new samples!
Key insight: Discriminative slightly better for classification, but generative can also generate data.
Interpolation between two points:
✓ Smooth transition across 10 steps
✓ All intermediate samples are valid
✓ No "jumps" or artifacts
Latent arithmetic:
Example: z_smiling - z_neutral + z_man = z_smiling_man
✓ Attributes transfer correctly
✓ Composition works as expected
Your implementation is complete when:
Common issues:
Verification:
# After E-step
assert np.allclose(responsibilities.sum(axis=1), 1.0), "Responsibilities must sum to 1"
# After M-step
assert np.allclose(weights.sum(), 1.0), "Weights must sum to 1"
Good sampling:
# Step 1: Choose component (categorical distribution)
k = np.random.choice(n_components, p=weights)
# Step 2: Sample from chosen Gaussian
sample = np.random.multivariate_normal(means[k], covariances[k])
Avoid:
# ❌ Don't sample component uniformly if weights are non-uniform
k = np.random.randint(n_components) # Wrong!
Effective plots:
Implement BIC to select optimal number of components:
def compute_bic(gmm, X):
"""
Compute BIC = -2 * log_likelihood + k * log(n)
where k = number of parameters
"""
# TODO: Implement BIC
pass
# Test with 1-10 components, choose best
Apply GMM to MNIST digits:
Implement conditional generation:
def conditional_sample(self, condition, n_samples):
"""
Generate samples conditioned on partial observations
Example: Generate y coordinate given x coordinate
"""
pass
Implement online/incremental EM for streaming data:
def online_update(self, new_batch):
"""
Update GMM parameters with new batch without storing all data
"""
pass
Completed Notebook: activity-09-introduction-to-generative-models.ipynb
Generated Visualizations:
Reflection (3-5 sentences):
Congratulations! You've built your first generative model.
Next Activity: Activity 10 - Build a Variational Autoencoder (VAE) for image generation
This activity is graded on:
Passing Grade: 70% or higher
Great work on your first generative model! 🎉🎨