00.02 · Core ML Concepts — Deep Dive

Level: Beginner
Pre-reading: 00 · ML Fundamentals · 00.01 · Supervised vs Unsupervised

Gradient Descent: How Models Learn

Gradient Descent is the algorithm that powers all modern machine learning. It's how we find weights that minimize loss.

Intuitive Explanation

Imagine you're standing on a hill in thick fog and want to reach the valley. You can't see far, but you can feel which direction is downhill and take a step that way. Repeat this process and you'll eventually reach the bottom.

Mathematical Formulation

The update rule for gradient descent:

Where: - \(w_t\) = weights at iteration \(t\) - \(\alpha\) = learning rate (step size) - \(\nabla L(w_t)\) = gradient of loss with respect to weights

Gradient = partial derivatives showing how loss changes with each weight:

Visualizing Gradient Descent

In 1D (single weight), loss is a curve. We start somewhere and slide downhill:

In 2D (two weights), loss is a landscape with valleys:

        ↓ Gradient points uphill
    *   * (step opposite direction)
   * ╲ * 
  *   V*  ← Direction of steepest descent
  *     *
   *   *  ← We step here

Learning Rate: The Tricky Hyperparameter

The learning rate \(\alpha\) controls step size. It's crucial:

Practical rule: Start with a small learning rate (0.001) and increase until loss starts oscillating. Then use a rate slightly smaller.

Batch Size

Do we update weights after seeing: - 1 example? (batch size = 1) - 32 examples? (batch size = 32) - All 10,000 examples? (batch size = 10,000)

Effects:

Intuition: Small batches give noisier gradient estimates but update more frequently. Large batches give cleaner estimates but update less frequently.

Variants of Gradient Descent

Batch Gradient Descent

Use entire dataset for each update:

• ✅ Smooth convergence
• ❌ Slow for large datasets
• ❌ Uses lots of memory

Stochastic Gradient Descent (SGD)

Update after each single example:

• ✅ Fast, frequent updates
• ❌ Noisy, can oscillate
• ❌ Hard to parallelize

Mini-Batch Gradient Descent

Update after each small batch (most common):

• ✅ Balanced — stable and fast
• ✅ Parallelizable
• ✅ Most practical approach

Optimization Algorithms: Beyond Basic Gradient Descent

Basic gradient descent can be slow. Modern algorithms add momentum and adaptive learning rates.

Momentum

Remember the direction you were moving and build momentum:

Intuition: Like a ball rolling down a hill — it gains speed in the downhill direction, helping it escape shallow local minima.

Adam (Adaptive Moment Estimation)

Combines momentum with adaptive learning rates per parameter:

Intuition: - \(m_t\) = exponential average of gradients (momentum) - \(v_t\) = exponential average of squared gradients (learning rate per parameter) - Parameters with large gradients get smaller updates; parameters with small gradients get larger updates

Why Adam is popular: - ✅ Works well with mini-batches - ✅ Handles sparse gradients - ✅ Adaptive learning rates eliminate manual tuning - ✅ Converges faster than vanilla SGD

Regularization: Preventing Overfitting

Overfitting is when a model memorizes training data instead of learning general patterns. Regularization adds constraints to prevent this.

L1 Regularization (Lasso)

Add penalty proportional to absolute values of weights:

Effect: - Encourages weights toward zero - Some weights become exactly zero (feature selection) - Results in sparse models

L2 Regularization (Ridge)

Add penalty proportional to squared values of weights:

Effect: - Discourages large weights - Spreads influence across features - Smoother, more stable models

L1 vs L2

Elastic Net = combination of L1 + L2, gets best of both.

Dropout

During training, randomly deactivate neurons with probability \(p\):

Why it works: - Prevents neurons from co-adapting (relying too much on each other) - Forces network to learn redundant representations - Acts like training many models and averaging them

During inference: Use all neurons but scale by \((1-p)\) to account for dropout.

Early Stopping

Monitor validation loss during training. Stop when it stops improving:

Why: - Training loss always decreases (model memorizes) - Validation loss eventually increases (overfitting) - Stop at the minimum validation loss

Batch Normalization

Normalize inputs to each layer to have mean 0 and standard deviation 1:

Benefits: - ✅ Stabilizes training (can use higher learning rates) - ✅ Reduces internal covariate shift - ✅ Acts as mild regularizer - ✅ Can sometimes eliminate need for dropout

When to use: Almost always in deep networks. It's become standard practice.

Hyperparameters vs Parameters

Hyperparameter tuning is the art of choosing values that lead to good generalization.

Common Hyperparameters

Strategy for tuning: 1. Start with default values from literature 2. Do quick coarse grid search (10x ranges) 3. Zoom in on good region 4. Fine-tune specific hyperparameters

Validation Metrics in Detail

Precision, Recall, and F1 for Classification

Confusion Matrix:

Metrics:

When to use each: - Accuracy: When classes are balanced and cost of false positives = cost of false negatives - Precision: When false positives are costly (email spam → wrong predictions annoying users) - Recall: When false negatives are costly (disease diagnosis → missing a disease is dangerous) - F1: When you want to balance both and classes are imbalanced

ROC-AUC for Ranking Models

Plot TPR vs FPR at different classification thresholds:

AUC = area under the ROC curve, measures how well model ranks positives above negatives.

Evaluation on Imbalanced Data

When one class dominates (e.g., 99% negative, 1% positive):

Don't use accuracy — a model predicting everything as negative would get 99% accuracy!

Instead use: - Precision-Recall curve — shows trade-off - F1 score — harmonic mean - AUC-ROC — threshold-independent

Or use: Class weights or oversampling/undersampling techniques.