Overfitting and Underfitting: How to Diagnose and Fix Both

Your model scores 95% on training data. Then it scores 60% on new data. What happened?

This gap has a name. Actually, it has two names, and they point to opposite problems. This article walks you through overfitting and underfitting: what causes each one, how to spot them fast, and exactly how to fix them. It’s the companion piece to Episode 55 of the Intelevo Machine Learning series on YouTube, so grab the video if you’d rather watch this play out visually.

Let’s get into it.

Why This Topic Matters

Every model you ever train will lean one way or the other. It will either miss the pattern because it’s too simple, or it will memorize the noise because it’s too complex. Rarely does a model land in the sweet spot on its first attempt.

That’s actually good news. Once you learn the signals, you can name the problem in seconds. Then you can apply the right fix instead of guessing. This single skill will save you hours of aimless tweaking.

The Analogy: Two Students, One Practice Test

Here’s a simple way to hold both ideas in your head at once.

Picture two students. Both prepare for an exam using the same set of practice questions. However, they prepare in very different ways.

The first student barely opens the book. They skim a few pages, and they never really absorb the concepts. As a result, they fail the practice test. Then they fail the real exam too, in the exact same way. They never learned the underlying pattern, so nothing clicks anywhere.

The second student takes a different approach. They memorize every single practice question, word for word. Consequently, they ace the practice test with a perfect score. But then the real exam arrives with slightly different questions, and everything falls apart. This student never learned the concept either. They just memorized answers instead.

Both students fail the real exam. Yet they fail for completely opposite reasons. That’s the whole story of overfitting and underfitting in one analogy.

What Is Underfitting?

Underfitting happens when your model is too simple for the pattern in your data. Think of it as trying to draw a straight line through a curve. No matter how you angle that line, it misses the shape.

Here’s how underfitting shows up in practice:

• Your model performs poorly on training data.
• It also performs poorly on test data.
• Train and test scores stay close together, but both sit low.

In short, an underfit model misses the pattern everywhere. It never learned enough to succeed anywhere, so both scores drop together.

What Is Overfitting?

Overfitting is the opposite disease. Here, your model becomes too complex for the data you gave it. Instead of learning the true pattern, it starts memorizing individual data points and their quirks.

Watch for these signs:

• Your model performs extremely well on training data.
• It performs noticeably worse on test data.
• A large gap opens up between the two scores.

Therefore, an overfit model looks fantastic on paper until you show it something new. Then the cracks appear immediately.

The Chart That Explains Everything

If you remember only one image from this topic, make it this one. Picture a graph with model complexity along the bottom, and error along the side.

As complexity rises, your training error keeps dropping. That makes sense: a more flexible model can always fit its own training data better. However, your test error behaves differently. It falls at first, hits a low point, and then climbs back up again.

This creates three zones:

1. Underfitting zone. On the left side, both errors sit high and close together.
2. Sweet spot. In the middle, test error reaches its lowest point.
3. Overfitting zone. On the right side, training error keeps shrinking, but test error climbs away from it.

That widening gap between train and test error is your clearest warning sign. Once you see it, you already know which direction to move.

A Fast Diagnostic Table

You don’t need fancy math to diagnose your model. Instead, just compare two numbers: your train score and your test score.

This table turns a vague feeling of “something’s wrong” into a clear next step. Use it every time you evaluate a new model.

How to Fix Underfitting

Once you confirm underfitting, every fix moves in the same direction. You need to give your model more room to bend. Try these approaches:

Add more features. Give your model more relevant information to work with. Sometimes the pattern exists in your data, but your current features simply don’t capture it.

Increase model complexity. Raise the polynomial degree, add layers to a neural network, or let a decision tree grow deeper. Each of these gives your model more flexibility.

Reduce regularization strength. If you’re already using Ridge, Lasso, or a similar technique, ease off the penalty. Too much regularization can push a model from a reasonable fit into underfitting territory.

Train for longer. Sometimes your model simply hasn’t finished learning yet. Give it more epochs or iterations before you judge its performance.

Each fix nudges your model toward more flexibility. Consequently, it becomes better equipped to capture the actual pattern in your data.

How to Fix Overfitting

Overfitting calls for the opposite response. Here, you rein complexity back in. Consider these fixes:

Collect more training data. With more examples, your model finds it much harder to memorize noise. Instead, it has to focus on the genuine signal, because the noise no longer repeats consistently.

Simplify the model. Lower the polynomial degree, prune a decision tree, or remove unnecessary features. A simpler model has fewer ways to memorize quirks.

Add regularization. Ridge, Lasso, and ElasticNet all penalize unnecessary complexity. As a result, they discourage your model from chasing every small fluctuation in the training data.

Stop training early, and validate often. Rather than training until the loss hits zero, monitor performance on a validation set. Stop as soon as validation performance starts to worsen, even while training performance keeps improving.

Notice how these fixes mirror the underfitting fixes exactly, just in reverse. That symmetry makes the whole topic much easier to remember.

A Key Idea Most Tutorials Skip: The Learning Curve

Before you rush off to collect more data, ask yourself a quick question: will more data actually help?

You can answer this by plotting error against training set size, not just model complexity. Practitioners call this chart a learning curve, and it reveals something crucial.

If your model suffers from high variance, the gap between train and test error shrinks as you add more training examples. In that case, more data genuinely helps.

However, if your model suffers from high bias, both curves stay high and close together no matter how much data you throw at them. The model itself is the bottleneck, not the size of your dataset.

This distinction matters a lot in practice. Otherwise, you might spend weeks collecting data that a high-bias model was never going to use effectively. A quick learning curve check saves you that wasted effort.

Watching the Fix Work in Python

Talking about fixes is one thing. Watching one work is far more convincing. Let’s revisit an overfitting example and apply a fix live.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_squared_error

poly = PolynomialFeatures(degree=15)
X_train_p = poly.fit_transform(X_train)
X_test_p  = poly.transform(X_test)

for name, model in [("No fix", LinearRegression()),
                     ("Ridge fix", Ridge(alpha=5.0))]:
    model.fit(X_train_p, y_train)
    train_err = mean_squared_error(y_train, model.predict(X_train_p))
    test_err  = mean_squared_error(y_test, model.predict(X_test_p))
    print(f"{name:10s} train={train_err:.2f}  test={test_err:.2f}")

Here’s what happens step by step. First, the code builds a degree-15 polynomial, which is a deliberately flexible and overfit-prone transformation. Next, it trains two separate models on the same transformed features.

The first model uses plain LinearRegression, with no fix applied at all. The second model wraps the same features in Ridge, using an alpha value of 5.0 to add regularization.

For each model, the code calculates mean squared error on both the training set and the test set, then prints both numbers side by side.

Without the fix, training error drops close to zero, yet test error stays stubbornly high. That gap is classic overfitting in action. Once Ridge regularization kicks in, test error actually drops, even though training error rises slightly. That trade-off is exactly what you want. You’ve sacrificed a small amount of training accuracy in exchange for a model that generalizes far better.

This is proof, not a guess. You can run this exact experiment on your own dataset and watch the same pattern emerge.

Underfitting vs. Overfitting: Quick Reference

Keep this comparison nearby whenever you evaluate a new model.

Notice the shared column: both problems produce a low test score. That’s exactly why you can’t diagnose the issue from test score alone. You always need to check the train score too.

The Four-Step Framework

Let’s bring everything together into one repeatable loop. Run through these four steps every time you train a new model.

1. Diagnose. Compare your train score to your test score.
2. Direction. A large gap points to high variance. Both scores sitting low points to high bias.
3. Adjust. Nudge complexity, data volume, or regularization in the direction the diagnosis suggests.
4. Recheck. Confirm your fix with cross-validation, rather than trusting a single lucky train-test split.

Once this loop becomes second nature, you’ll stop guessing altogether. Instead, you’ll approach every model with a clear, methodical process.

Common Mistakes to Avoid

Even after you learn the theory, a few habits can quietly sabotage your diagnosis. Watch out for these.

Judging a model from test score alone. A low test score never tells the whole story by itself. Always pair it with your train score before you decide anything. Otherwise, you might apply an overfitting fix to a model that’s actually underfitting, and your results will only get worse.

Tuning on the test set. If you keep adjusting your model based on test performance, you slowly leak information from that test set into your decisions. Consequently, your test score stops representing truly unseen data. Use a separate validation set, or rely on cross-validation, to keep your final test set honest.

Trusting a single train-test split. One split can mislead you through pure chance. A lucky split might hide a variance problem, while an unlucky one might exaggerate it. Cross-validation smooths out that noise and gives you a far more reliable diagnosis.

Assuming more data always helps. As you saw earlier with learning curves, more data only helps when variance is the actual problem. If bias is the issue, extra data just delays the moment you realize your model needs to change, not your dataset.

Forgetting to recheck after a fix. A single improved score doesn’t confirm success. Run cross-validation again after every adjustment. That way, you confirm the fix generalizes, rather than just getting lucky on one evaluation.

Avoiding these five habits will make your diagnosis far more reliable, and it will save you from chasing the wrong fix entirely.

Frequently Asked Questions

Can a model suffer from both overfitting and underfitting at once? Not in the same sense described here, but a model can definitely be poorly tuned in ways that mix both symptoms across different subsets of your data. Generally, though, one problem tends to dominate at any given complexity level.

Which problem is worse, overfitting or underfitting? Neither is inherently worse. Both produce unreliable predictions. However, overfitting can be more dangerous in production, since your training metrics might look great even while your model fails silently on new data.

Do neural networks face these same issues? Yes, absolutely. Neural networks are especially prone to overfitting, since they contain so many parameters. That’s exactly why techniques like dropout, early stopping, and weight decay exist.

How much of a train-test gap counts as a problem? There’s no single universal threshold, since it depends on your dataset and your tolerance for error. Still, a consistently widening gap across multiple cross-validation folds is a reliable warning sign, regardless of the exact numbers involved.

Watch the Full Video

This article summarizes Episode 55 of the Intelevo Machine Learning series. The video walks through every diagram, the full code demo, and a live walkthrough of the four-step framework in action.

If this helped you, please like the video, subscribe to Intelevo, and drop a comment with your questions. I read every single one, and your feedback genuinely shapes future episodes.

Up next, in EP56, we move from manually fixing models to letting the machine search for the best settings automatically with Grid Search. See you there.