Data Preparation Exercise

What You'll Learn

In this hands-on exercise, you'll gain practical experience with:

• :emoji: Data Cleaning: Handle missing values, duplicates, and inconsistent formats
• :wrench: Data Preprocessing: Scale features, encode variables, and handle imbalanced data
• :bar_chart: Data Splitting: Properly divide your dataset for training and testing
• :bulb: Best Practices: Learn industry-standard techniques for preparing data

Total Time Required: 90-120 minutes

Access the exercise template here: Data Preparation Exercise Template

Exercise Overview

:computer: Part One: Data Cleaning (30-40 minutes)

In this section, you will:

1. Handle missing values in the dataset

   • Identify missing data patterns
   • Choose appropriate filling strategies
   • Implement your chosen method

2. Remove duplicate entries

   • Find duplicate rows
   • Decide which duplicates to keep
   • Clean the dataset

3. Fix inconsistent data formats

   • Standardize date formats
   • Normalize text casing
   • Fix numerical inconsistencies

4. Clean text data

   • Remove extra whitespace
   • Handle special characters
   • Standardize text formatting

:bulb: Common Challenge: When handling missing values, consider the data type and context. Numerical data might use mean/median, while categorical data might use mode or a special category.

:computer: Part 2: Data Preprocessing (30-40 minutes)

You will learn to:

1. Scale numerical features
   • Apply StandardScaler or MinMaxScaler
   • Understand when to use each method
   • Validate the scaling results
2. Encode categorical variables
   • Use OneHotEncoder for nominal data
   • Apply LabelEncoder for ordinal data
   • Handle unknown categories
3. Handle imbalanced datasets
   • Identify class imbalance
   • Apply SMOTE or undersampling techniques
   • Verify the balanced distribution
4. Create new features
   • Engineer meaningful features from existing data
   • Combine features for better insights
   • Document your feature creation logic tip Best Practice: Always fit your scalers and encoders on the training data only, then transform both training and test sets. This prevents data leakage!

:computer: Part 3: Data Splitting (20-30 minutes)

This section covers:

1. Split data into training and testing sets

   • Import train_test_split from sklearn.model_selection
   • Separate features (X) from target (y)
   • Apply the split function

2. Configure split parameters

   • Set test_size to 0.2 (20% for testing)
   • Use random_state=42 for reproducibility
   • Understand stratification for imbalanced data

3. Validate the split results

   • Check the shapes of train and test sets
   • Verify the proportion of split
   • Ensure no data leakage

Implementation Task:

• Split your preprocessed data using the exact parameters: test_size=0.2 and random_state=42
• Print the shapes of all resulting arrays (X_train, X_test, y_train, y_test)

:warning: Warning Important: Your submission must use the exact parameters specified above for consistency in grading.

:emoji: Exercise Submission

:information_source: Submission Deadline : Submit your completed notebook within the allocated class time or as specified by your instructor.

Please submit your work through this link: Exercise Submission Form

Submission Guidelines

1. Complete all questions in each part
2. Ensure your code runs without errors from start to finish
3. Follow the provided instructions exactly:
   • Use test_size=0.2 for data splitting
   • Use random_state=42 for reproducibility
4. Save your notebook with your name: DataPrep_YourName.ipynb
5. Submit the completed notebook through the form

:warning: Warning Before Submitting:

• Restart your kernel and run all cells to ensure everything works
• Double-check that all outputs are visible
• Verify you've used the correct parameters

:wrench: Troubleshooting

Common Issues and Solutions

1. Import Errors

   python

   # Make sure to install required packages
   !pip install pandas numpy scikit-learn imbalanced-learn
   

2. Memory Issues

   • Work with a sample of data if the full dataset is too large
   • Clear variables you no longer need: del variable_name

3. Scaling Errors

   • Ensure you're only scaling numerical columns
   • Check for infinite or NaN values before scaling

4. Encoding Issues

   • Handle unknown categories in test set
   • Consider using handle_unknown='ignore' parameter

:bulb: Tip Need Help? If you encounter issues not listed here, ask your instructor or post in the class discussion forum.