Custom Object Detection with YOLOv8

:information_source: Project Overview Difficulty Level: Intermediate
Estimated Time: 3-4 hours
Skills Practiced:

• Training YOLO models
• Working with image datasets
• Object detection implementation
• Model deployment
• Real-time video processing

In this project, you will use your knowledge in Computer Vision & Object Detection to create a rock-paper-scissors detector. The algorithm we will use to carry out the Detection is YOLOv8.

In the first part, we will write the codes for training the object detector using this dataset from Roboflow. This will be done using Google Colabs. In the second part, we will use the trained model from Part 1 to make predictions using images and videos. This will be done in Visual Studio Code.

In the advanced challenge, we will attempt to deploy the project onto streamlit application.

If you face technical issues with implementing YOLOv8 in VSCode, you may refer to these guidelines to complete the project in Google Colabs.

Visual Project Roadmap

graph TB
    A[Start: Project Setup] --> B[Part 1: Model Training]
    B --> C[Dataset Exploration]
    C --> D[Train YOLOv8 Model]
    D --> E[Download Trained Model]
    E --> F[Part 2: Implementation]
    F --> G[Image Detection]
    F --> H[Webcam Detection]
    G --> I[Advanced: Streamlit App]
    H --> I
    I --> J[Project Complete!]
    
    style A fill:#e1f5fe
    style B fill:#fff3e0
    style D fill:#ffecb3
    style F fill:#c8e6c9
    style I fill:#f3e5f5
    style J fill:#a5d6a7

Project Overview

Rock Paper Scissors is a hand game, usually played between 2 players, in which the players will simultaneously form one of the three shapes with their hands: Rock, Paper, Scissors.

The dataset we will be using to train our object detector contains about 1700 images of rock, paper and scissors hand shape. It originates from the website Roboflow which contains a lot of different image dataset, some comes together with annotation.

Here are some examples of images in the dataset:

To learn more about the dataset, feel free to visit the link here.

:bulb: Why This Project? This project combines machine learning theory with practical implementation. You'll not only train a model but also deploy it in real-world scenarios - a crucial skill for AI developers!

Part One: Model Training (Google Colab)

:warning: Before You Start Make sure you have:

• A Google account to access Google Colab
• Stable internet connection (model training requires continuous connection)
• About 30 minutes for the training process

Phase One: Setup

1. Make a copy of the template file found here and rename the copied file as "P6: Custom Object Detection - Part 1".

:bulb: Pro Tip Always make a copy of the template instead of editing the original. This allows you to restart fresh if needed!

2. Run the code here to install Ultralytics to Google Colabs.

   python

   # Install ultralytics for YOLOv8
   !pip install ultralytics
   

   What's happening: This installs the Ultralytics library which contains the YOLOv8 implementation we'll use for object detection.

3. Run the code here to download the dataset onto your colab file. Wait for the download process to complete before proceeding to the next step.

   python

   import gdown
   gdown.download('https://drive.google.com/uc?id=1OWa_HT1ZXmOo2bnXjmOZI47Cvam_NyU5', 'rock_paper_scissors.zip')
   

   What's happening: This downloads a pre-prepared dataset containing 1700+ labeled images of rock, paper, and scissors hand gestures.

   If the download fails, you may import the dataset "rock_paper_scissors.zip" into your file. For collab users, run the following codes to import json files:

   python

   from google.colab import files
   upload = files.upload()
   

:dart: Milestone Checkpoint One: You should now have Ultralytics installed and the dataset downloaded in your Colab environment.

Phase 2: Explore the Data

In this phase, we will explore and take a look at what kind of data we are working with. (No codes needed)

:bulb: Why Data Exploration? Understanding your data is crucial! It helps you verify the dataset is loaded correctly and gives insight into what the model will be learning from.

1. Run all the codes given here to conduct the exploration process.

2. Check out the shape of the training images. If the codes are ran properly you should obtain the following answers:

   makefile

   Shape: (10, 640, 640, 3)
   

   What this means:

   • 10 = Number of sample images loaded
   • 640, 640 = Image dimensions (width x height in pixels)
   • 3 = Color channels (RGB)

3. Take a look at the first image in the dataset.

:dart: Milestone Checkpoint 2: You've successfully explored the dataset and understand the image dimensions.

Phase 3: Object Detection Model Training

:warning: Training Time Alert The training process will take approximately 10-15 minutes. Don't close your browser or let your computer sleep during this time!

1. Import YOLO from the Ultralytics package.

2. Run the codes to save the path to the root directory of data.yaml to ROOT_DIR.

   python

   # Getting the root directory of data.yaml
   import os
   ROOT_DIR = os.path.abspath(os.curdir)
   

   What's happening: This gets the current directory path where your data.yaml file (dataset configuration) is located.

3. Train and validate the performance of the YOLO model. Note that this step might take some time (about 10 minutes).

   python

   # Train and validate YOLO model
   model = YOLO()
   model.train(data=ROOT_DIR+"/data.yaml", epochs=10)
   model.val()
   

   What's happening:

   • YOLO() creates a new YOLOv8 model
   • model.train() trains the model for 10 epochs (complete passes through the dataset)
   • model.val() validates the model's performance on test data

:bulb: Training Progress Watch the training progress! You'll see:

• Loss values decreasing (good sign!)
• mAP (mean Average Precision) scores increasing
• Training examples being processed

:dart: Milestone Checkpoint 3: Your model is now trained! You should see validation metrics showing how well it performs.

Phase 4: Download Model

:bulb: Keep Your Model Safe! The trained model is your hard work! Make sure to download it successfully before closing Colab.

1. Run the following 2 lines of codes to save the model into a zipped file called "model.zip".

   python

   import locale
   locale.getpreferredencoding = lambda: "UTF-8"
   

   What's happening: This fixes potential encoding issues when creating the zip file.

   python

   !zip -r /content/model /content/runs/detect/train
   

   What's happening: This compresses your entire trained model (weights, configuration, etc.) into a single zip file.

2. You can then run the following codes to download the zip file.

   python

   # Download model onto device
   from google.colab import files
   files.download('/content/model.zip')
   

   What's happening: This triggers a download of your model.zip file to your local computer.

:warning: Download Issues? If the download doesn't start:

• Check your browser's download permissions
• Try right-clicking the file in Colab's file panel and selecting "Download"
• Make sure pop-ups aren't blocked

:dart: Milestone Checkpoint 4: You now have your trained model downloaded and ready for Part 2!

Part 2: Object Detection Implementation (VSCode)

:information_source: Part 2 Overview Now we'll use our trained model to detect rock, paper, and scissors in real-world scenarios!

Phase One: Project Folder Setup

:warning: Prerequisites Before starting Part 2, ensure you have:

• VSCode installed on your computer
• Python 3.7+ installed
• The model.zip file from Part 1
• A webcam (for Phase 3)

1. Download the template from this link.

2. Extract the template inside your Program G Folder. Then open the folder using your VSCode.

3. Move and unzip the downloaded model.zip file from part 1 into your template folder.

:bulb: Unzipping Tips

• Windows: Right-click -> "Extract All"
• Mac: Double-click the zip file
• Make sure to extract INTO your project folder, not beside it!

4. The final file structure of your exercise folder should look like this:

5. To finish the setup, run the following command in your terminals to install all the required packages. (Remarks: You may skip this step if you had installed Ultralytics previously)

   1. For Windows Users:

      bash

      py -m pip install -r requirements.txt
      

   2. For MacOS Users:

      bash

      python3 -m pip install -r requirements.txt
      

:warning: Installation Errors? If you encounter errors:

• Make sure you're in the correct directory
• Try pip install -r requirements.txt (without py/python3)
• Check your Python installation with python --version

:dart: Milestone Checkpoint 5: Your VSCode environment is now set up with the trained model!

Phase 2: Object Detection (Image)

Please refer to Chapter 17: Object Detection with YOLOv8 to complete the following codes. Write your codes in the file object_detection(image).py.

:bulb: Building Step by Step This phase teaches you to process static images first - it's easier to debug than real-time video!

1. Import YOLO from Ultralytics.

2. Import OpenCV and Math packages

3. Load the trained rock-paper-scissors model. This model is located under the path "content/runs/detect/train/weights/best.pt". (Remarks: You may copy the following codes to complete this step)

   python

   model = YOLO(model="content/runs/detect/train/weights/best.pt")
   

   What's happening: This loads your custom-trained model (not the default YOLO model).

4. Load the model's labels and store them as class_names.

5. Load image from 'image1.jpg' and store it as img.

6. Resize the image to the dimension (1280 x 720).

   Why resize? Consistent image size ensures reliable detection performance.

7. Detects objects in img and store them as results.

8. Draw the bounding boxes and labels on the image. Note that the font size in cv.putText should be set to One.0.

:bulb: Visualization Tips

• Use different colors for each class (rock, paper, scissors)
• Make bounding boxes thick enough to see clearly
• Position labels so they don't overlap with the boxes

9. Show the image.

:dart: Milestone Checkpoint 6: You should see your image with bounding boxes around detected hands!

Phase 3: Object Detection (Webcam)

Please refer to Chapter 17: Object Detection with YOLOv8 to complete the following codes. Write your codes in the file object_detection(webcam).py.

:warning: Webcam Permissions Your computer may ask for camera permissions. Make sure to allow VSCode/Python to access your webcam!

1. Repeat step 1 - 4 from phase 2.

2. Capture video from a webcam and store it as webcam.

3. Within the while loop:

   1. Capture frames from a webcam.
   2. Quit when there are no more frames.
   3. Run object detector to detect the objects in the image.
   4. After detection, draw the bounding boxes around the image.
   5. Display resulting frame.
   6. Stop the video playback when the Q key is pressed.

:bulb: Performance Tips If the webcam feed is laggy:

• Reduce the frame size before processing
• Skip every other frame
• Lower the detection confidence threshold

4. Release the webcam when everything is done.

:bulb: Testing Your Detector Try these hand positions:

• Hold your hand steady for 2-3 seconds
• Use good lighting
• Keep your hand at arm's length from the camera
• Try different angles!

:dart: Milestone Checkpoint 7: Your real-time rock-paper-scissors detector is working! Play with friends to test it out!

Debugging Tips

:information_source: Common Issues and Solutions Having trouble? Here are solutions to frequent problems:

Model Not Loading?

• Check the file path: content/runs/detect/train/weights/best.pt
• Ensure model.zip was extracted properly
• Verify the folder structure matches the template

No Detections Showing?

• Verify your model trained successfully (check Part 1 output)
• Try lowering the confidence threshold
• Ensure proper lighting in your test images/video

OpenCV Errors?

# If cv2 not found:
pip install opencv-python

# If camera not working:
# Try different camera indices: 0, 1, 2...
webcam = cv2.VideoCapture(1)  # Instead of 0

Performance Issues?

• Resize frames before processing:

frame = cv2.resize(frame, (640, 480))

• Process every nth frame instead of all frames

Advanced Challenges

:bulb: Ready for More? These challenges will push your skills further! Each one builds on what you've learned and adds real-world applications.

Advanced Challenge One: Web Application Deployment

We had successfully implemented and developed a simple object detector using YOLOv8 that can detect rock, paper and scissors hand gestures. As an advanced challenge, try to implement the object detector in a streamlit application, where the user can upload their own pictures, then your object detector will show what is detected on the picture. You may refer to the official streamlit docs to learn how to use the file uploader.

:bulb: Streamlit Starter Code

python

import streamlit as st
from ultralytics import YOLO
import cv2

st.title("Rock Paper Scissors Detector")
uploaded_file = st.file_uploader("Choose an image...", type=["jpg", "png"])

if uploaded_file is not None:
    # Your detection code here
    pass

Advanced Challenge 2: Live Camera Integration

Based on challenge 1, try to modify the code to accept images from webcam by taking a picture instead of the file uploader. You may find the documentation to the webcam_capture of streamlit here.

:bulb: Enhancement Ideas

• Add a game mode where users play against the computer
• Keep score of wins/losses
• Add sound effects for each detection

Advanced Challenge 3: Build Your Own Custom Detector

If you manage to reach this challenge, this means you are ready to face the ultimate challenge, which is to build your very own custom object detector and host it onto streamlit. You may use the following guidelines as a general references on how you should do it.

:warning: Project Ideas Choose a dataset that interests you:

• Pet breed detector (cats vs dogs)
• Fruit classifier (apples, oranges, bananas)
• Sports equipment detector
• Traffic sign recognition

Step-by-Step Guide:

• Head onto Roboflow's website and sign up for an account.

• Search for a suitable dataset that you wish to use for your object detector. (Remarks: For practice purposes, it would be much preferable if you choose a small dataset with about 100 - 500 images to complete the challenge).

• Download the selected dataset in YOLOv8 format:

• Unzip the downloaded file to get the folders that look something like this:

• Open the folder train and move all the labels in the labels folder into the images folder. (Note that if you select a large dataset, the copy process will take some time.)

• Zip the folders together into a zip file, then upload it to your Google Colabs. (Note that if you select a large dataset, the upload process will take some time).

• From here onwards, refer to Part 1 & 2 to complete your object detector.

:bulb: Success Tips

• Start with a simple, well-labeled dataset
• Train for more epochs (20-30) for better accuracy
• Test your model thoroughly before deployment
• Share your creation with classmates!

Extension Challenges

:star2: Challenge: Real-time Game Implementation

Create a playable Rock-Paper-Scissors game where:

• The computer randomly selects its move
• Your detector identifies the player's hand gesture
• Score is kept and winner is announced
• Add a countdown timer for each round

:star2: Challenge: Multi-Player Detection

Modify your detector to:

• Detect multiple hands simultaneously
• Identify which player made which gesture
• Determine the winner when two players play

:star2: Challenge: Gesture Recognition System

Extend beyond Rock-Paper-Scissors:

• Add more hand gestures (thumbs up, peace sign, etc.)
• Create a sign language letter detector
• Build a hand-gesture-controlled application

:information_source: Congratulations! You've completed a full machine learning project from data to deployment! This project demonstrates:

• Data preparation and model training
• Real-world implementation
• Web application deployment
• Creative problem-solving

Keep exploring and building amazing AI applications! :rocket: