Word Representations - Embeddings & Neural Networks

:dart: Learning Objectives

By the end of this lesson, you will:

• Understand how computers turn words into numbers
• Explain what word embeddings are and why they're amazing
• See why simple methods don't work well for understanding language
• Learn how neural networks work like a brain
• Discover how AI learns word meanings from reading text
• Identify different types of word relationships

:emoji: From Words to Numbers: The Challenge

:information_source: The Big Problem : Computers only understand numbers, but we speak in words! To help AI understand language, we need to turn words into numbers without losing their meaning.

Think about it - when you read the word "cat," you instantly picture a furry animal that says "meow." But a computer just sees letters. We need a smart way to help computers understand what words mean!

:emoji: The Problem with Simple Approaches

One-Hot Encoding: Too Simple!

Let's start with the simplest idea - giving each word its own special spot in a list.

Example - Like Assigned Seats:

• "cat" = [1, 0, 0, 0, 0] <- Cat sits in seat 1
• "dog" = [0, 1, 0, 0, 0] <- Dog sits in seat 2
• "animal" = [0, 0, 1, 0, 0] <- Animal sits in seat 3
• "car" = [0, 0, 0, 1, 0] <- Car sits in seat 4
• "vehicle" = [0, 0, 0, 0, 1] <- Vehicle sits in seat 5

:memo: Why This Doesn't Work Well:

• The computer thinks "cat" and "dog" are as different as "cat" and "car" :emoji:
• It doesn't know that cats and dogs are both animals
• We'd need millions of seats for all English words!
• The computer can't tell which words have similar meanings

# Example: One-hot encoding representation
vocabulary = ["cat", "dog", "animal", "car", "vehicle"]

def one_hot_encode(word, vocabulary):
    """Convert a word to one-hot encoding"""
    vector = [0] * len(vocabulary)
    if word in vocabulary:
        index = vocabulary.index(word)
        vector[index] = 1
    return vector

# Examples
cat_vector = one_hot_encode("cat", vocabulary)
dog_vector = one_hot_encode("dog", vocabulary)
print(f"Cat: {cat_vector}")
print(f"Dog: {dog_vector}")

:star2: Word Embeddings: A Better Solution

:information_source: Word Embeddings : Smart number lists where similar words get similar numbers. Think of it like a magical map where words that mean similar things live in the same neighborhood!

Instead of giving each word just one seat, embeddings give each word a whole set of scores that describe what it means.

Watch: Storage and Embeddings Explained

:sparkles: Key Properties of Word Embeddings

One. Semantic Similarity - Words That Mean Similar Things Stick Together!

:bulb: Words with similar meanings live close to each other in the embedding space - like best friends sitting together at lunch!

Examples of Word Neighbors:

• "king" and "queen" are neighbors (both royalty)

• "cat" and "dog" are closer than "cat" and "car"

• "happy" and "joyful" are practically roommates!

2. Analogical Relationships - Word Math Magic! 🪄

You can actually do math with words! Check out these amazing examples:

Famous Word Math:

• King - Man + Woman = Queen :emoji:

• Paris - France + Italy = Rome :emoji:️

• Walking - Walked + Swimming = Swam :emoji:

3. Dimensional Meaning - Each Number Tells a Story

Think of each dimension like a personality trait for words:

What Each Dimension Might Measure:

• Dimension 1: Is it alive? (animals vs. objects)

• Dimension 2: How big is it? (elephant vs. ant)

• Dimension 3: How does it feel? (happy vs. sad)

• Dimension 4: How formal is it? (goodbye vs. bye)

:emoji: How Neural Networks Learn Word Meanings

Neural networks learn like you do - by reading lots and lots of text! They figure out what words mean by looking at the words around them.

The Secret: You Are Known by Your Friends! :emoji:

note The Distributional Hypothesis: "You shall know a word by the company it keeps" - J.R. Firth

This fancy saying means: Words that hang out together usually have similar meanings!

Let's See It in Action:

• "The cat sat on the mat" vs. "The dog sat on the mat"
  • Both cat and dog can "sit on mats" = They're similar!
• "I drink coffee" vs. "I drink tea"
  • Both are things we drink = They're related!
• "She feels happy" vs. "She feels joyful"
  • Both describe good feelings = They mean almost the same thing!

🪟 Context Windows - Looking at Word Neighborhoods

The AI looks at words near each other (like looking through a window at your neighbors).

Example: "The quick brown fox jumps over the lazy dog"

For the word "fox" (window size = 2 words):

• The AI sees: ["quick", "brown", "jumps", "over"]
• It learns: "Fox" hangs out with action words and descriptive words!

# Example: Context window extraction
def extract_context(sentence, target_word, window_size=2):
    """Extract context words around a target word"""
    words = sentence.split()
    target_index = words.index(target_word)
    
    start = max(0, target_index - window_size)
    end = min(len(words), target_index + window_size + 1)
    
    context = words[start:target_index] + words[target_index+1:end]
    return context

sentence = "The quick brown fox jumps over the lazy dog"
context = extract_context(sentence, "fox", window_size=2)
print(f"Context for 'fox': {context}")

:emoji:️ Neural Network Architecture for Word Embeddings

The Building Blocks of a Neural Network

Think of a neural network like a super-smart factory that turns words into understanding!

:emoji: Input Layer - The Front Door

This is where words enter the network (usually as one-hot encoding at first).

:wrench: Hidden Layers - The Magic Workshop

These layers do the hard work! They:

• Find patterns in the data
• Learn what makes words similar or different
• Transform simple inputs into smart representations

:emoji: Output Layer - The Exit

This gives us the final answer, which could be:

• Language Modeling: "What word comes next?"
• Sentiment Analysis: "Is this happy or sad?"
• Translation: "How do you say this in Spanish?"

:dart: Word2Vec: A Popular Embedding Method

Word2Vec is like a super-popular recipe for making word embeddings. It has two flavors:

:emoji: Skip-gram Model - "Guess My Friends!"

• What it does: Takes one word and guesses its neighbors
• Example: Given "fox" -> Guess ["quick", "brown", "jumps", "over"]
• Like playing: "If I say fox, what words usually come with it?"

:emoji: Continuous Bag of Words (CBOW) - "Guess Who's Missing!"

• What it does: Takes neighbor words and guesses the middle word
• Example: Given ["quick", "brown", "jumps", "over"] -> Guess "fox"
• Like playing: "Fill in the blank: The quick brown ___ jumps"

:emoji: Training Process - Teaching the Network

:bulb: Training is like teaching a student - start with random guesses and get better over time!

1. :emoji: Initialize: Start with random numbers for each word

2. :emoji:️ Feed Forward: Send words through the network

3. :x: Calculate Loss: Check how wrong our guess was

4. ⬅️ Backpropagate: Learn from mistakes and adjust

5. :emoji: Repeat: Keep practicing until we get it right!

python

# Simplified example of embedding training concept
import numpy as np

class SimpleWordEmbedding:
    def __init__(self, vocab_size, embedding_dim):
        """Initialize random word embeddings"""
        self.vocab_size = vocab_size
        self.embedding_dim = embedding_dim
        # Random initialization
        self.embeddings = np.random.randn(vocab_size, embedding_dim)

    def get_embedding(self, word_index):
        """Get embedding vector for a word"""
        return self.embeddings[word_index]

    def similarity(self, word1_index, word2_index):
        """Calculate cosine similarity between two words"""
        vec1 = self.embeddings[word1_index]
        vec2 = self.embeddings[word2_index]

        dot_product = np.dot(vec1, vec2)
        magnitude1 = np.linalg.norm(vec1)
        magnitude2 = np.linalg.norm(vec2)

        return dot_product / (magnitude1 * magnitude2)

# Example usage
vocab_size = 10000
embedding_dim = 300
model = SimpleWordEmbedding(vocab_size, embedding_dim)

# Get similarity between two words (by their indices)
similarity_score = model.similarity(42, 108)
print(f"Similarity: {similarity_score}")

:emoji: Types of Word Relationships

Word embeddings are amazing at finding different kinds of relationships between words!

:emoji: Semantic Relationships - Meaning Connections

:emoji: Synonymy - Words That Mean the Same Thing

• happy ~= joyful ~= cheerful :emoji:

• big ~= large ~= huge :emoji:

• car ~= automobile ~= vehicle :emoji:

:zap: Antonymy - Opposites Attract!

• hot <-> cold :fire::emoji:️
• big <-> small :emoji::emoji:
• happy <-> sad :emoji::emoji:

:emoji: Hypernymy/Hyponymy - Parent and Child Words

• animal (parent) -> dog, cat, bird (children)

• flower (parent) -> rose, tulip, daisy (children)

• fruit (parent) -> apple, banana, orange (children)

:memo: Syntactic Relationships - Grammar Buddies

:runner: Part of Speech - Words That Work the Same Way

• Action words (verbs): run, walk, jump
• How words (adverbs): quickly, slowly, carefully
• Describing words (adjectives): red, blue, green

:emoji: Grammatical Forms - Same Word, Different Outfit

• walk -> walked -> walking

• good -> better -> best

• I -> me -> my

:emoji: Cultural and Contextual Relationships

:emoji: Occupational - Job Families

• Medical: doctor, nurse, surgeon
• Education: teacher, professor, student
• Food Service: chef, waiter, restaurant

:emoji:️ Geographic - Place Connections

• Cities: Paris, London, Tokyo

• Nature: mountain, river, ocean

• Directions: north, south, east, west

:emoji:️ Visualizing Word Embeddings

Since word embeddings live in super high dimensions (like 100-300!), we need special tricks to see them on a flat screen. note The Challenge: Imagine trying to draw a 3D cube on paper - now imagine doing that with 300 dimensions! :emoji:

:art: t-SNE - The Artist's Method

t-SNE squishes high dimensions down to 2D while keeping similar words close together. It's like making a flat map of Earth - not perfect, but useful!

:bar_chart: PCA - The Mathematician's Method

PCA finds the most important patterns and focuses on those. It's like taking a photo from the best angle to show the most information.

# Example: Visualizing word embeddings (conceptual)
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

def visualize_embeddings(embeddings, words, method='tsne'):
    """Visualize word embeddings in 2D"""
    if method == 'tsne':
        reduced = TSNE(n_components=2).fit_transform(embeddings)
    
    plt.figure(figsize=(12, 8))
    plt.scatter(reduced[:, 0], reduced[:, 1])
    
    for i, word in enumerate(words):
        plt.annotate(word, (reduced[i, 0], reduced[i, 1]))
    
    plt.title('Word Embeddings Visualization')
    plt.show()

# Usage would require actual embedding data
# visualize_embeddings(word_vectors, word_list)

:emoji: Contextual Embeddings: Beyond Static Representations

:emoji: The Problem with One-Size-Fits-All Embeddings

Traditional embeddings give each word the same numbers every time. But wait - many words have different meanings!

:bulb: Multiple Personalities: Some words are like actors playing different roles in different sentences!

Words with Multiple Meanings:

• "bank" :emoji: (where you keep money) vs. "bank" :emoji:️ (side of a river)

• "bat" :emoji: (flying animal) vs. "bat" :emoji: (for hitting baseballs)

• "spring" :emoji: (season) vs. "spring" :wrench: (bouncy metal coil)

:emoji: Contextual Embeddings - Smart, Flexible Word Meanings

Modern AI (like BERT and GPT) creates different embeddings based on how the word is used!

Same Word, Different Meanings:

• "I went to the bank to deposit money" -> bank = :emoji: (financial)

• "We sat by the river bank" -> bank = :emoji:️ (geographical)

:rocket: Applications of Word Embeddings

:emoji: Real-World Uses

:emoji::emoji: Sentiment Analysis - Detecting Feelings in Text

Figures out if someone is happy, sad, or neutral from their words.

python

# Example: Using embeddings for sentiment analysis
def sentiment_from_embeddings(text_embeddings):
    """Classify sentiment using word embeddings"""
    # Average word embeddings in the text
    avg_embedding = np.mean(text_embeddings, axis=0)

    # Use the embedding to predict sentiment
    # (In practice, this would involve a trained classifier)
    sentiment_score = classify_sentiment(avg_embedding)
    return sentiment_score

:emoji: Machine Translation - Breaking Language Barriers

Converts text from one language to another by finding matching meanings across languages.

:mag: Information Retrieval - Smart Search

Finds documents that match what you're looking for, even if they use different words!

:dart: Recommendation Systems - "You Might Also Like..."

Suggests similar items based on word similarity (like Netflix recommendations!).

:art: Creative Applications

:emoji: Analogy Completion - Word Puzzles!

• King is to man as queen is to ____? (woman!)
• Paris is to France as Tokyo is to ____? (Japan!)

:video_game: Word Association Games

Find words related to a starting word - great for brainstorming!

:emoji:️ Creative Writing Helper

Suggests better words, finds synonyms, or helps continue your story!

:dart: Practical Exercise: Understanding Word Relationships

Let's play with word relationships to see how embeddings work!

:emoji:️ Activity One: Semantic Clustering Challenge

Your Mission: Group these words by what they mean:

• dog, car, cat, truck, hamster, bicycle, goldfish, motorcycle tip Think about what these things have in common!

Answer:

• :emoji: Animals: dog, cat, hamster, goldfish
• :emoji: Vehicles: car, truck, bicycle, motorcycle

:emoji: Activity 2: Analogy Reasoning

Complete these word puzzles:

• King : Queen :: Man : ____ (Answer: Woman)
• Swim : Swimming :: Run : ____ (Answer: Running)
• Happy : Happiness :: Sad : ____ (Answer: Sadness)

:emoji: Activity 3: Context Detective

How is "bow" different in each sentence?

• "She took a bow after her performance" (bending to thank audience)
• "He used a bow and arrow for hunting" (curved weapon)
• "The ship's bow cut through the waves" (front of ship)

:warning: Challenges and Limitations

:emoji: Bias in Embeddings - When AI Learns Bad Habits

:memo: Important: AI learns from human text, so it can pick up human biases too!

Examples of Bias:

• Gender bias: AI might think "doctor" = male, "nurse" = female (not true!)

• Racial bias: Some names get unfairly linked to negative words

• Cultural bias: Western ideas show up more than others

:emoji: Out-of-Vocabulary Words - "I've Never Seen This Before!"

AI struggles with words it hasn't learned:

• New slang: "That's bussin!" (means really good)
• Names: Your friend's unique name
• Special terms: New technology words
• Typos: "teh" instead of "the"

:computer: Computational Requirements - It Takes a Lot!

• Memory: Needs lots of computer memory to store all word vectors

• Power: Requires powerful computers to train

• Time: Can take days or weeks to learn from all that text!

:emoji: Future Directions

:emoji: Multilingual Embeddings - One AI, Many Languages!

Future embeddings will understand multiple languages at once, making it easy to translate and communicate globally!

:iphone: Dynamic Embeddings - AI That Keeps Learning

Embeddings that update themselves when new words appear (like the latest TikTok slang!).

:mag: Interpretable Embeddings - Understanding the Magic

Making it easier to see exactly what each dimension means - no more mystery numbers!

:books: Key Terms and Vocabulary

:information_source: Quick Reference : Important words you learned today!

• Word Embedding: Smart number lists that capture word meanings
• One-Hot Encoding: Simple method where each word gets one spot (too simple!)
• Context Window: The neighborhood of words around a target word
• Semantic Similarity: How close two words are in meaning
• Vector Space: The magical multi-dimensional world where embeddings live
• Cosine Similarity: Math way to measure how similar two word vectors are
• Dimensionality Reduction: Squishing high dimensions down so we can see them
• Skip-gram: "Guess my friends" - predicts context from a word
• CBOW: "Fill in the blank" - predicts word from context

:emoji: Summary

Word embeddings are one of the coolest breakthroughs in AI! They help computers understand language by turning words into smart numbers.

:star2: What You Learned Today:

• Words -> Numbers: Computers need numbers to understand our words
• Smart Neighborhoods: Similar words live close together in embedding space
• Context Matters: AI learns meanings by looking at word neighbors
• Word Math Works: King - Man + Woman = Queen (mind-blowing!)
• Multiple Meanings: Modern AI handles words with different meanings
• Real Applications: From translation to helping you write better! tip Remember: Word embeddings are like giving computers a dictionary where every word has coordinates instead of definitions!

:rocket: Next Steps

Ready for more? In our next lesson, we'll explore attention mechanisms and output generation - the super-powers that make ChatGPT and other modern AI so amazing!

:emoji: Practice Prompts

Test your understanding with these fun challenges:

1. Word Neighborhood Game: List 5 words that would be neighbors to "pizza" in embedding space. Why did you choose them?

2. Analogy Creator: Create your own word analogy like "King:Queen::Man:Woman". Make it creative!

3. Context Detective: Write two sentences using the word "park" with completely different meanings. How would AI tell them apart?

4. Bias Buster: Think of a job. What words might AI wrongly associate with it? How could we fix this?

5. Future Thinker: If you could teach AI one new word relationship, what would it be and why?