Data Cleaning with Pandas Activity

What you'll learn

• How to identify and handle missing data in datasets
• Techniques for removing duplicate entries
• Methods to convert data types for proper analysis
• Best practices for cleaning and standardizing data
• How to export cleaned datasets for future use

:computer: Coding Exercise

Time estimate: 45-60 minutes

Getting Started

1. Open the Google Colab notebook: Data Cleaning Exercise
2. Important: Make a copy of the file (File -> Save a copy in Drive)
3. Work in your personal copy, NOT the template file

:warning: Important Always create a copy of the Colab notebook before starting. This ensures:

• Your work is saved to your Google Drive
• You don't accidentally modify the template
• You can return to your work later

Data Cleaning Steps with Pandas

Step One: Import Libraries (5 minutes)

1. Import the pandas library as pd
2. Import the numpy library as np
3. Run the cell to confirm imports are working

Step 2: Load Your Dataset (5 minutes)

1. Use pd.read_csv() to load the provided dataset
2. Store it in a variable called df
3. Print a success message when loaded

:bulb: If you get a "file not found" error, check that the CSV file path is correct in your Colab environment.

Step 3: Explore Your Data (10 minutes)

1. Use df.head() to view the first 5 rows
2. Use df.columns to list all column names
3. Use df.info() to see data types and missing values
4. Use df.shape to check the dataset dimensions
5. Take notes on what needs cleaning

Step 4: Remove Unnecessary Columns (5 minutes)

1. Identify columns that won't be used in analysis
2. Use df.drop() with the column names
3. Set axis=1 to indicate you're dropping columns
4. Verify columns are removed with df.columns

Step 5: Handle Duplicate Values (5 minutes)

1. Check for duplicates using df.duplicated().sum()
2. View duplicate rows with df[df.duplicated()]
3. Remove duplicates using df.drop_duplicates()
4. Verify removal by checking the shape again tip Common Challenge Sometimes you may want to keep duplicates based on certain columns only. Use the subset parameter in drop_duplicates() to specify which columns to check.

Step 6: Handle Missing Values (10 minutes)

1. Check missing values with df.isnull().sum()
2. Visualize missing data patterns (optional)
3. Choose a strategy:
   • Drop rows: df.dropna()
   • Fill with mean: df.fillna(df.mean())
   • Fill with mode: df.fillna(df.mode())
4. Apply your chosen strategy
5. Verify missing values are handled

Step 7: Fix Data Types (5 minutes)

1. Check current types with df.dtypes
2. Convert numeric columns: pd.to_numeric()
3. Convert date columns: pd.to_datetime()
4. Convert categories: .astype('category')
5. Verify conversions with df.dtypes

:bulb: Tip Use errors='coerce' in conversion functions to handle invalid values gracefully.

Step 8: Fix Incorrect Values (5 minutes)

1. Check for negative values in columns that should be positive
2. Use boolean indexing to find incorrect values
3. Replace or correct these values
4. Standardize text values (e.g., uppercase, lowercase)
5. Verify corrections are applied

Step 9: Export Cleaned Data (5 minutes)

1. Use df.to_csv('cleaned_data.csv', index=False)
2. Download the file from Colab
3. Verify the exported file opens correctly

Troubleshooting Guide

Common Issues and Solutions

Problem: "No module named pandas" error

• Solution: Run !pip install pandas in a cell first

Problem: Data types not converting properly

• Solution: Check for special characters or spaces in your data
• Use errors='coerce' to handle problematic values

Problem: Memory errors with large datasets

• Solution: Use chunksize parameter when reading CSV
• Work with a sample of data first using df.sample()

Problem: Cleaned data not saving

• Solution: Check file permissions in Colab
• Try downloading directly after creating the file

:information_source: Helpful Resources

• Pandas Documentation on Data Cleaning
• Colab File Management Guide

---

:emoji: Submission Instructions

:warning: Before You Submit :white_check_mark: Ensure all code cells have been run successfully :white_check_mark: Your cleaned dataset is exported and downloadable :white_check_mark: You've made a copy of the Colab notebook :white_check_mark: All steps are completed with comments explaining your approach

Submit Your Work

1. Save your Colab notebook (File -> Save)
2. Get the shareable link (Share -> Copy link)
3. Submit your exercise here: :link: Submission Form

:information_source: Submission Checklist

• Colab notebook link
• Cleaned CSV file
• Brief summary of cleaning steps taken
• Any challenges faced and how you solved them