Gradient Descent Explained: How Machines Learn to Improve (Batch, Stochastic, Mini-Batch)

Every machine learning model needs a way to learn from its mistakes. But how does that actually happen under the hood? This gradient descent explained guide answers that question in plain English. You’ll walk away understanding batch, stochastic, and mini-batch gradient descent, plus the one formula that ties them all together.

This article is the companion piece to Episode 38 of the Intelevo Machine Learning series on YouTube. If you prefer to watch and listen, check out the video first. Then, come back here to review the concepts at your own pace.

Let’s get started.

Why Gradient Descent Matters

You’ve likely trained a regression model already. You called .fit(), and moments later, your model produced predictions. But something important happened behind that single line of code. Your model didn’t just guess randomly. Instead, it searched for the best possible parameters, step by step.

That search process has a name: gradient descent. It’s the engine behind nearly every model you’ll ever train, from simple linear regression to deep neural networks. Once you understand it, a lot of machine learning suddenly makes more sense.

So, let’s build your intuition first. Then, we’ll add just enough math to make the picture complete.

The Big Analogy: Lost on a Foggy Mountain

Imagine you’re standing on a mountain, and thick fog surrounds you. You can’t see the valley below. However, you can feel the ground sloping beneath your feet. That single sensation is enough to guide you downhill.

Here’s how the process unfolds, step by step:

1. You start somewhere on the mountain. This spot represents your model’s initial guess for its parameters.
2. You feel the slope beneath you. This slope represents the gradient.
3. You take a step downhill. This step represents one update to your model’s parameters.
4. You repeat this process. Eventually, the ground feels flat, and you’ve reached the bottom.

That’s the entire idea behind gradient descent explained in one simple story. Every technical detail we cover next builds directly on this picture. So, keep this mountain in mind as we go.

What Are We Actually Measuring? The Loss Function

Before you can walk downhill, you need to know what “height” means on this mountain. That height comes from something called the loss function.

Put simply, the loss function scores how wrong your model’s predictions are right now. Higher ground means higher loss, so your predictions are worse. Lower ground means lower loss, so your predictions are better.

In other words, the mountain is the loss function. Every possible combination of model parameters corresponds to a specific location on this terrain. Your goal is simple: find the lowest point.

Here’s the only formula you need for this part:

MSE = (1/n) * Σ (actual − predicted)²

This is Mean Squared Error, or MSE. It takes the difference between the actual value and the predicted value, squares it, and averages that result across all your data points. The bigger your miss, the higher the ground climbs. That’s it.

The Gradient: Just a Fancy Word for Slope

The term “gradient” sounds intimidating, but don’t let it fool you. It simply means slope. Once you internalize that, half the intimidation disappears.

Consider three scenarios:

• Steep slope: You’re far from flat ground, so you take a confident, larger step.
• Gentle slope: You’re getting close to the bottom, so you slow down and step carefully.
• Flat ground: The slope equals zero, so you’ve arrived at the minimum.

This behavior leads us to the single update rule that powers gradient descent:

new guess = old guess − (step size × slope)

In mathematical notation, we write this as:

θ = θ − α · ∇J(θ)

Let’s break that down quickly:

• θ (theta) represents your model’s parameters, or your current position on the mountain.
• α (alpha) represents the learning rate, or your stride length.
• ∇J(θ) represents the gradient, or the slope you feel beneath your feet.

That’s the whole update rule. Every version of gradient descent we discuss below uses this exact formula. The only thing that changes is how much data you check before taking each step.

The Learning Rate: Your Stride Length

Before we move to the three main types of gradient descent, let’s address one crucial dial: the learning rate.

Think of the learning rate as your stride length while walking downhill. Get it wrong, and your journey turns messy fast.

If your learning rate is too small, you take tiny steps. Eventually, you’ll reach the bottom, but the process feels painfully slow.

If your learning rate is too large, you overshoot the valley entirely. You bounce between the walls, or worse, you fly off the mountain completely. In practice, this shows up as a loss value that grows instead of shrinking.

If your learning rate is well-tuned, you take confident yet controlled steps. As a result, you settle smoothly into the valley without excessive delay or instability.

So, what’s the practical takeaway? Start with a moderate learning rate. Then, if your loss bounces around instead of settling, shrink that rate until training stabilizes.

Three Ways to Check the Slope

Up to this point, we’ve described checking the entire mountain before every step. But in practice, you have three different strategies for sensing the slope. Let’s explore each one.

Batch Gradient Descent

Batch gradient descent surveys the entire mountain before taking a single step. In other words, it looks at all of your training data, computes the average slope, and only then moves forward.

This approach offers real advantages. First, it produces a very stable, smooth, and direct path downhill. However, it also comes with real costs. Because it scans your entire dataset every time, each step takes longer. Additionally, it demands more memory since it needs the whole dataset loaded at once.

Batch gradient descent works best with small, clean datasets, where precision matters more than raw speed. Think of it like polling every single villager before deciding which way the road slopes. Thorough, yes. Fast, not always.

Stochastic Gradient Descent (SGD)

Stochastic gradient descent, or SGD, takes the opposite approach. Instead of checking the whole mountain, it checks just one random data point, then steps immediately. It grabs an example, senses the ground right there, and moves. Then, it repeats this process with a fresh random example.

This method runs very fast, since updates happen instantly. On the other hand, the resulting path looks noisy, zig-zagging on its way downhill. Still, SGD stays light on memory, since it only needs one example at a time.

SGD shines with huge or streaming datasets, where speed matters more than a perfectly smooth path. Picture asking just one villager and immediately walking in that direction. Quick, yes. Occasionally jittery, also yes.

Mini-Batch Gradient Descent

Mini-batch gradient descent strikes a balance between the two extremes above. Rather than scanning everything or checking just one point, it samples a small batch, perhaps 32 or 64 examples. It averages their slope, takes a step, then repeats with the next batch.

As a result, you get balanced speed. It runs faster than batch gradient descent, yet stays calmer than SGD. Its path remains mostly smooth, with just a mild wobble along the way. Better still, mini-batch gradient descent works efficiently on GPUs, which explains why it dominates modern deep learning.

In practice, mini-batch gradient descent serves as the default choice for most real-world projects. Picture asking a small trail group of hikers instead of one person or an entire village. Quick enough, and noticeably steadier than asking just one.

Comparing All Three Methods

Let’s place these three methods side by side, so the differences become crystal clear.

Notice the pattern here. As you move from batch to stochastic, you trade stability for speed. Mini-batch gradient descent then recovers much of that lost stability, while keeping most of the speed gains. That’s precisely why it earned its spot as the industry standard.

Seeing Gradient Descent in Python Code

Theory helps, but code makes everything click. Let’s implement gradient descent from scratch, using nothing but NumPy.

import numpy as np

X = np.array([1, 2, 3, 4, 5])
y = np.array([7, 10, 13, 16, 19])   # true pattern: y = 4 + 3x

w, b, lr = 0.0, 0.0, 0.01            # starting point and stride length

for step in range(1000):
    y_pred = w * X + b
    error  = y_pred - y
    dw = (2/len(X)) * np.dot(error, X)   # slope with respect to w
    db = (2/len(X)) * np.sum(error)      # slope with respect to b
    w -= lr * dw                          # step downhill
    b -= lr * db

print(w, b)   # approaches 3.0, 4.0

Let’s walk through this line by line.

First, we import NumPy and set up a tiny dataset. Our data follows a clear pattern: y equals 4 plus 3x. Naturally, we want our model to discover this pattern on its own.

Next, we initialize w and b, our weight and bias, both at zero. This starting point represents an arbitrary spot on our mountain. We also set our learning rate, lr, to 0.01. This value controls our stride length throughout training.

Then, we loop for 1,000 steps. Inside this loop, several things happen in sequence. First, we calculate y_pred, our current prediction, using the formula w times X plus b. Next, we calculate the error, which equals our prediction minus the actual value.

After that, we calculate dw and db. These two values represent the slopes with respect to w and b. They come directly from the calculus behind Mean Squared Error, which conveniently simplifies to a dot product and a sum. Finally, we update w and b by subtracting the learning rate multiplied by their respective slopes. This update represents our step downhill.

After all 1,000 iterations complete, printing w and b reveals numbers very close to 3 and 4. In other words, our simple loop successfully learned the exact pattern hiding inside the data. No external library did the heavy lifting here. Just a handful of lines, and a little patience.

What the Descent Actually Looks Like

If you could watch these three methods in action, side by side, you’d notice distinct visual signatures.

Batch gradient descent takes smooth, confident, and relatively few big steps. Its path traces a direct line straight to the bottom. Stochastic gradient descent, by contrast, looks jittery. It takes many tiny steps, bouncing around as it works its way downhill. Mini-batch gradient descent lands somewhere in between. It shows a gentle wobble, yet maintains steady overall progress.

All three methods reach the same destination eventually. However, they clearly take very different paths to get there. Keep this image in mind whenever you’re debugging a model that trains unpredictably. The method you chose might explain the pattern you’re seeing.

Key Takeaways

Let’s bring everything together into one clean summary.

• Gradient descent means walking downhill on the loss surface until the ground turns flat.
• The gradient represents the slope you feel. It tells you which direction leads downhill.
• The learning rate represents your stride length. Aim for a value that’s neither too big nor too small.
• Batch gradient descent surveys the whole mountain each step. It stays stable, though it moves slowly.
• Stochastic gradient descent checks one data point at a time. It moves fast, though it stays noisy.
• Mini-batch gradient descent checks a small group at a time. It serves as the everyday default for good reason.

Once you see gradient descent through this lens, the concept stops feeling abstract. Instead, it becomes an intuitive, repeatable process: sense the slope, take a step, and repeat until you reach flat ground.

What’s Next: Evaluating Regression Models

Now that you understand how a model learns, a natural question follows. How do you know if it learned well? That question leads directly into our next episode, EP39: Evaluating Regression Models.

In that episode, we’ll cover MAE, MSE, RMSE, and R-squared. Each metric offers a different lens for judging your model’s performance. Together, they’ll give you the confidence to evaluate any regression model you build going forward.

Watch the Full Video

This gradient descent explained article summarizes the key ideas from Episode 38 of the Intelevo Machine Learning series. For the complete walkthrough, including live explanations and additional visuals, watch the full video on the Intelevo YouTube channel.

If this article helped clarify gradient descent for you, consider subscribing to the channel. Also, drop a comment on the video and let us know which method, batch, stochastic, or mini-batch, you’ll reach for first in your own projects. Your feedback genuinely shapes future episodes, so don’t hold back.

See you in EP39, where we’ll finally answer the question: was your model actually any good?