Handling Missing Values: How to Clean Gaps in Your Dataset Without Losing Your Data

Every real dataset has gaps. A sensor drops a reading. A customer skips a survey question. A form field stays empty. These gaps are missing values, and they show up in almost every dataset you will ever touch.

This article is the companion piece to Episode 22 of the Intelevo Machine Learning series on YouTube. Watch the video first if you prefer a visual, spoken walkthrough. Then use this article as your reference guide. Bookmark it, take notes from it, and come back to it whenever you clean a new dataset.

By the end of this guide, you will know exactly what a missing value is, why it appears, and which fix to reach for in any situation. Let’s get started.

Why Missing Values Matter More Than You Think

Picture a busy restaurant. A waiter takes down each order on a pad: spice level, table size, wait time. On a calm night, every box gets filled in. On a hectic night, though, a few boxes stay blank. The pad still goes to the kitchen. The meal still gets cooked. Nobody panics over one missing box.

Now hand that same pad to a machine learning model instead of a chef. The model does not shrug off a blank box the way a person does. Instead, it does one of two things. Either it refuses to run at all, or it quietly treats the blank as a zero and produces a prediction nobody asked for.

That difference is the whole reason this topic matters. A model cannot guess what a blank box meant on its own. You have to decide first, and you have to decide on purpose. This guide shows you exactly how.

What a Missing Value Actually Looks Like

In pandas, every empty cell carries the same label: NaN, short for “Not a Number.” Consider this small table of restaurant orders:

Three columns here each contain one NaN cell. Notice something important: NaN does not mean zero. It does not mean average, either. It simply means “unknown here.” Keep that distinction in mind, because it shapes every decision that follows.

Every gap you will ever clean starts out exactly like this. It is just an ordinary NaN cell, nothing more mysterious than that.

Step One: Count the Gaps Before You Touch Anything

Before you fix a single value, take stock of the whole dataset. This step matters because you cannot repair what you have not measured. One line of pandas code handles the counting for you:

df.isnull().sum()

This line checks every column and returns the number of missing entries in each one. For example, it might return this:

spice_level      1
size             1
wait_time_min    1
satisfaction     0

Now you know exactly where to focus. Skipping this step is like a chef trying to cook without first checking which ingredients ran out. Always count first.

Why Do Gaps Happen in the First Place?

Not every blank box tells the same story. Understanding the reason behind a gap helps you choose the right fix. Broadly, three patterns show up again and again.

First, some gaps are just forgotten. A waiter gets busy and skips a box for no particular reason connected to the order itself. Statisticians call this pattern “random,” because the gap has nothing to do with the value that is missing.

Second, some gaps follow a pattern. On a rushed night, the wait time never gets logged at all. Here, the gap depends on how busy the shift was, not on the individual customer. This is a “patterned” gap, and it is worth noticing because it can quietly bias your results if you ignore it.

Third, some gaps carry meaning on their own. A table that hated the meal might simply refuse to leave a satisfaction score. In this case, the missing value itself is a clue about the outcome.

You do not need to memorize formal statistical terms to act wisely here. Instead, ask yourself one simple question: does the reason for this gap relate to the answer I am trying to predict? That single question will guide most of your cleaning decisions.

Option One: Dropping the Gaps

Once you know where your gaps live, you have two main paths forward. The first path is dropping.

Dropping works well when only a handful of rows have gaps out of a much larger dataset. In that case, the simplest fix is to remove those rows and move on. Think of it as tossing a single torn order pad out of five hundred. It is not worth rebuilding, so you bin it and continue with the rest.

Here is the code:

# drop rows with any gap
clean = df.dropna()

# drop a column instead
clean = df.drop(columns=["wait_time_min"])

The first line removes every row that contains at least one missing value. The second line removes an entire column instead, which makes sense when that column is unreliable across the board.

However, dropping comes with a cost. Every row you remove is a row you lose forever. This approach works fine for a small number of gaps in a large dataset. It becomes risky once gaps pile up, because you start throwing away good data along with the bad.

Option Two: Filling the Gaps

The second path is filling, also known as imputation. This approach makes more sense once gaps become common, because dropping every affected row would waste too much good data.

Instead of deleting the row, you write in a sensible stand-in value. Suppose nobody logged the wait time for one particular table. Rather than leave that cell blank, you fill it with tonight’s typical wait time. A reasonable guess beats an empty box every time.

Here is how that looks in code:

avg_wait = df["wait_time_min"].mean()
df["wait_time_min"] = df["wait_time_min"].fillna(avg_wait)

This line calculates the average wait time across the dataset, then fills every missing entry with that number. As a result, you keep every row in your dataset. That said, every filled value adds a small amount of noise, so reserve this method for columns where gaps show up often, not for rare, one-off cases.

Choosing the Right Average: Mean, Median, or Mode

Filling a gap raises an immediate follow-up question: which average should you actually use? Three simple options cover almost every situation you will face.

Mean equals the sum divided by the count. Use it for typical numbers without extreme outliers, such as wait time on a normal night.

Median is the middle value once you sort all the numbers. Use it when a few unusually large numbers would drag the mean too high. For example, one table that waited three hours would badly skew a mean calculation, but the median stays stable.

Mode is simply the most common value in the column. Use it for categories rather than numbers. If a size value goes missing, filling it with “Medium” makes sense, since that is the most frequently ordered size.

In code, all three options slot into the exact same line:

df["wait_time_min"].fillna(df["wait_time_min"].median())

Swap in .mean() or .mode()[0] depending on your situation, and the logic stays identical.

Dropping Versus Filling: A Quick Comparison

At this point, you might wonder which method wins overall. The honest answer is neither, because each one suits a different situation. Here is a side-by-side view to make the choice easier.

Use this table as a quick decision-making tool. If gaps are rare, lean toward dropping. If gaps are widespread, lean toward filling. Either way, make the choice deliberately rather than by default.

A Quick Gut-Check

Try this scenario before moving on. Suppose one column in your dataset is ninety percent missing. Almost every value you fill in would be a pure guess. What should you do?

The right move is to drop the column entirely. When a column has almost no real data left, filling it does not rescue it. Instead, filling simply dresses up guesses as if they were real information. Recognizing this situation early saves you from building a model on shaky ground.

You Already Make These Decisions Every Day

This idea is not new to you, even if the terminology sounds unfamiliar. Consider a few everyday examples.

On a job application, an optional field left blank does not stop the form from working. It simply gets skipped or defaulted. In a survey, people skip questions they do not want to answer, yet analysts still summarize the rest of the responses. Weather stations lose sensor readings for an hour at a time, so forecasters interpolate from nearby stations instead. On a medical form, a patient might forget an old surgery date, so a doctor works with what is known rather than what is missing.

In every case, someone makes a judgment call about a gap without treating it as a crisis. Machine learning simply asks you to make that same judgment call on purpose, with code instead of instinct.

Four Pro Tips for Cleaner Results

As you apply these techniques to your own projects, keep these four habits in mind. They will save you from subtle, hard-to-catch mistakes.

First, fill gaps after splitting your data into training and test sets, never before. Calculate the mean or median using only the training data. If you fill gaps before splitting, you leak information from the test set into training, which quietly inflates your model’s apparent performance.

Second, keep a “was missing” flag. Add a new column, such as wait_time_missing, with a value of 1 or 0. This way, your model can learn that the fact a value was missing might matter just as much as the value itself.

Third, check the pattern before you choose a method. Running df.isnull().sum() or plotting a quick heatmap tells you whether gaps scatter randomly or cluster together. This check takes seconds and prevents poor decisions later.

Fourth, avoid dropping your way to an empty dataset. If every column has at least one gap somewhere, calling dropna() without care can quietly delete almost your entire dataset. Always check how many rows survive before committing to that method.

Putting It All Together: A Full Code Walkthrough

Here is a complete example that combines everything covered so far. Run this code as a final check before training any model.

import pandas as pd

df = pd.read_csv("orders.csv")

# 1. see where the gaps are
print(df.isnull().sum())

# 2. flag missingness before filling
df["wait_missing"] = df["wait_time_min"].isnull().astype(int)

# 3. fill the numeric gap with the median
df["wait_time_min"] = df["wait_time_min"].fillna(
    df["wait_time_min"].median())

# 4. confirm the data is clean
print(df.isnull().sum().sum())
# Out: 0

Notice the order of these four steps. First, count. Next, flag. Finally, fill. Following this sequence ensures you never lose the fact that a value was once missing, even after you replace the blank itself. If the final print statement returns zero, your dataset is ready for the next stage of your pipeline.

Key Takeaways

Let’s recap everything this guide covered, since each point builds directly toward better, more reliable models.

• Missing values show up as NaN in pandas, and NaN simply means “unknown here,” not zero and not average.
• Gaps generally fall into three patterns: random, patterned, or meaningful. Ask whether the gap relates to your prediction target.
• Dropping rows or columns works well when gaps are rare and your dataset is large.
• Filling gaps works well when gaps are common, using the mean, median, or mode depending on the situation.
• A “was missing” flag preserves valuable information that a simple fill would otherwise erase.
• Always clean your data after splitting into training and test sets, never before.

Handling missing values does not require complex statistics or heavy math. It requires a clear process: count, understand, decide, and act. Apply that process consistently, and messy, real-world data stops feeling intimidating.

Watch the Full Video and Keep Learning

This article summarizes Episode 22 of the Intelevo Machine Learning series. For the full walkthrough, complete with visuals and live code demonstrations, watch the video on YouTube. If you found this guide useful, please like the video, subscribe to the channel, and leave a comment with your questions or feedback. Your comments genuinely shape what gets covered in future episodes.

Up next, Episode 23 tackles outliers and duplicates. Some orders get copied twice by mistake, and some wait times show up as three hours because a clock glitched somewhere along the way. Learning to catch both issues protects your model from getting warped by bad data, so stay tuned.

Thank you for reading, and see you in the next episode.