Data Splitting Strategies and Cross-Validation in Machine Learning

You just fixed your imbalanced dataset. Your model looks ready. But here’s a question that trips up almost every beginner: how do you actually know if your model works?

This question sits at the heart of cross-validation in machine learning, and it’s the exact topic we cover in Episode 28 of the Intelevo Machine Learning Series. If you haven’t watched the video yet, press play above before you read on. This article walks through the same ideas, so you can revisit them anytime, take notes, and copy the code without pausing a video.

Let’s get into it.

Why a Fair Test Matters

Imagine a teacher who hands out the exact exam questions as homework the night before. Every student aces the test. But did anyone actually learn the material? No. The score looks great, yet it tells you nothing real.

Machine learning models fall into the same trap. Train a model, then test it on the very data it just memorized, and you get an illusion of success. The accuracy looks perfect. However, that number tells you how well the model memorized, not how well it will handle tomorrow’s new data.

This is why every reliable machine learning workflow starts with an honest split between what a model learns from and what it gets judged on. And that judgment needs to happen more than once, so it isn’t just luck.

Meet the Exam: One Analogy for the Whole Topic

Throughout this article, think of your model as a student preparing for a big exam. This one analogy ties every concept together, so keep it in mind as you read.

First, the student studies from a textbook. That’s your training set — the material the model learns from, again and again, until patterns start to sink in.

Next, the student takes a mock test. That’s your validation set, used to check progress and fine-tune how the student studies.

Then comes the final exam. That’s your test set — questions the student has never seen before, and the only fair measure of real understanding.

Finally, instead of relying on just one mock test, the student retakes several different practice exams. That’s cross-validation, and it means one lucky, or unlucky, paper never decides the final grade.

Keep this analogy close. Every section below builds directly on it.

The Basics: Splitting Data Into Train and Test

Before any training begins, you need to divide your dataset. Set a portion aside, and never touch it during training.

A typical split looks like this:

• 80% training data — what the model learns from
• 20% test data — held back and used only once, at the very end

Why does this matter so much? First, it prevents memorization, since the model can’t simply “remember” the answers. Second, it simulates the future, because test data stands in for new, real-world cases the model hasn’t met yet. Third, it keeps you honest, since you never get to peek at the test set while you’re still building the model.

Common ratios include 80/20 and 70/30, and the right choice depends on how much data you have. Either way, always shuffle your rows before splitting. Otherwise, one set might end up looking completely different from the other.

One Split Isn’t Quite Enough

A simple train-test split works fine, right up until you start tuning your model. Maybe you try different settings. Maybe you compare several versions and pick the best one. Every time you check the test set to make that decision, you quietly leak information into it.

So here’s the fix: add a third set. Split your data into training, validation, and test sets, like this:

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.4, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42)

# 60% train   20% validation   20% test

This three-way split adds a validation set, which acts as your practice exam. You tune your model against it as much as you like. Meanwhile, the test set stays sealed until the very end, so your final grade stays honest.

But here’s a new question: what if that one validation split happened to be an easy one, or an unlucky one?

The Luck of the Draw

Picture two equally prepared students taking different mock tests. Their practice scores can look wildly different, and that difference has nothing to do with skill. It comes down to which questions happened to show up.

The same thing happens with a single validation split. If the validation rows happen to contain simple examples, your score looks better than the model deserves. On the other hand, if those rows happen to be tricky edge cases, that same model suddenly looks worse than it deserves.

Here’s the one idea to remember from this section: a single validation split is a single roll of the dice. To trust the score, you need to average many rolls instead of trusting just one. That single idea leads directly to cross-validation.

Cross-Validation: The Fix

Cross-validation doesn’t rely on one lucky or unlucky split. Instead, it rotates through several splits, so every row eventually gets a fair turn as both study material and test question.

The process breaks down into three simple steps:

1. Split into folds. Divide your training data into k equal-sized chunks, called folds.
2. Rotate the test fold. Train on k−1 folds, then validate on the one fold you left out. Repeat this until every fold has had its turn as the test set.
3. Average the scores. Combine all k scores into a single, reliable average. That average becomes your cross-validated score.

This process sounds complex at first, but once you see it laid out visually, it clicks immediately.

How K-Fold Cross-Validation Works, Visually

Let’s walk through five-fold cross-validation, since it’s the most common choice.

Picture five rounds. In round one, fold one plays the test set, while folds two through five train the model. In round two, fold two takes a turn as the test set, and the rest go back to training. This pattern continues through round five.

By the end, every single fold has served as the test set exactly once. Nothing gets wasted, and no fold gets an unfair advantage over another. Once you collect all five scores, say 0.91, 0.88, 0.93, 0.89, and 0.90, you simply average them. In this example, that gives you a final, trustworthy score of 0.902.

Notice how this number reflects five honest mock tests, rather than one lucky attempt. That’s the entire point.

Cross-Validation in Python: Three Lines, Not Thirty

Here’s the part that surprises most beginners: you don’t need to write this rotation logic by hand. Scikit-learn handles it in a single function call.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
scores = cross_val_score(model, X_train, y_train, cv=5)

print(scores)
# [0.91  0.88  0.93  0.89  0.90]
print(scores.mean())
# 0.902  <- the trustworthy number

Setting cv=5 alone gives you five honest mock-test scores, with no manual splitting required. Report the mean as your headline number. If you also want to show how stable your model is, report the spread between the scores too.

Stratified K-Fold: Keeping Folds Fair for Imbalanced Data

Remember the rare-fraud problem from the last episode? It shows up again right here.

Plain random folds can accidentally dump most of the rare class into just one or two folds. As a result, the remaining folds get almost no examples of that class to learn from, and your cross-validation scores become unreliable.

This is exactly why Stratified K-Fold exists. It keeps the same class ratio inside every single fold, so each round of testing sees a fair, realistic mix of both classes.

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf)

Make Stratified K-Fold your default choice whenever your classes are imbalanced. It costs you nothing extra, and it removes an entire category of bad luck from your results.

How Many Folds Should You Use?

More folds give you a steadier average. However, more folds also mean more training runs, so there’s always a tradeoff to consider.

For most projects, k=5 gives you the best balance between speed and reliability. Save Leave-One-Out for genuinely small datasets, since it trains one model per row and quickly becomes expensive.

The Number One Mistake: Data Leakage in Preprocessing

Right after “never resample before you split” from the last episode, here’s the second most common mistake: scaling or filling in missing values using statistics from the whole dataset, before you split it.

The wrong order: scale the whole dataset, then split it into train and test.

This quietly lets your scaler learn the test set’s mean and spread. That’s a small leak, but it’s a real one, and it lets future information sneak into your training process.

The right order: split first, fit your scaler only on the training data, then apply that same scaler to your validation and test sets.

scaler.fit(X_train)                        # learn mean/spread from train only
X_train_scaled = scaler.transform(X_train)
X_test_scaled  = scaler.transform(X_test)   # apply the same rule, never refit

This way, your test data stays a true stranger to the model, exactly like real-world new data will be. Treat this rule the same way you treat the resampling rule from last episode: split first, always.

Where This Shows Up in the Real World

These ideas extend far beyond a single homework project.

On Kaggle, every competition leaderboard gets scored on a hidden test split, which is exactly the discipline covered in this article. In clinical trials, researchers split and validate patient groups in rounds before trusting any treatment claim. In A/B testing, comparing two product versions relies on this same honest-split thinking. At scale, companies choose between dozens of models using cross-validated scores, not a single lucky run.

In other words, once you understand this topic, you start noticing it everywhere.

Frequently Asked Questions

Is cross-validation always necessary?

Not always, but it helps far more often than it hurts. For quick experiments or massive datasets, a single validation split can be enough. However, whenever you’re comparing models, tuning hyperparameters, or working with a smaller dataset, cross-validation gives you a far more trustworthy picture of real performance.

Does cross-validation replace the test set?

No, and this trips up a lot of beginners. Cross-validation happens on your training data, and it helps you tune and compare models fairly. Your test set still stays sealed until the very end, untouched by any of this rotation. Think of cross-validation as multiple mock exams, and the test set as the one final exam that decides the real grade.

What’s the difference between K-Fold and Stratified K-Fold?

Regular K-Fold splits your data into folds randomly, without checking class balance. Stratified K-Fold checks the class ratio first, then builds folds that preserve that same ratio throughout. For balanced datasets, the difference barely matters. For imbalanced datasets, like the fraud example from the last episode, Stratified K-Fold becomes essential.

Quick Recap

Let’s tie everything together:

• Split your data correctly. Use train, validation, and test sets, and keep the test set sealed until the very end.
• Trust cross-validation. Rotate through k folds, then average the scores instead of trusting one lucky split.
• Keep your folds fair. Use Stratified K-Fold whenever your classes are imbalanced.
• Avoid leakage. Fit scalers and encoders on training data only, and never let test data shape your preprocessing.

Put these four habits together, and you get a score you can actually stand behind.

What’s Next

Now that you know how to split and validate data honestly, the next episode puts that knowledge to work on real, messy data. Episode 29 covers working with real-world datasets from Kaggle and the UCI Machine Learning Repository, where the cleanup begins before any splitting can happen.

If this article helped you, watch the full video for the visual walkthrough of five-fold cross-validation, complete with the diagram that makes it click instantly. And if you’re new here, subscribe to Intelevo so you don’t miss the rest of this Machine Learning Series.

Got a question about train-test splits or cross-validation? Drop it in the comments. I read every single one.