You just trained a model. It reports 95% accuracy. You feel great.
But wait. Is that number actually good? Or is it lying to you?
This is exactly the question this episode answers. Welcome to Part 2 of the Intelevo Capstone series. In EP105, we framed the problem and collected our data. Now, we take the next step: training candidate models and learning proper machine learning model evaluation. This skill separates confident beginners from reliable practitioners. Without it, you’re just guessing. With it, you know exactly why your model works, and when it doesn’t.
Watch the full video here: Intelevo Youtube Channel
Let’s dig in.
Why a Single Model Never Proves Anything
Here’s the core problem. You train one model. It fits. It predicts. It looks decent. So you ship it.
But a single model, trained once, tells you nothing about whether it’s actually good. Training accuracy looks impressive almost by default. Models are built to minimize error on the data they see. So of course they look good on that same data.
The real test happens later, when new data arrives. That’s where the gap shows up. A model that memorized its training set instead of learning general patterns will fail quietly. And by then, it’s already in production.
This is why machine learning model evaluation isn’t optional. It’s the difference between a model you can trust and a number you hope is true.
The Cooking Competition Analogy
Think about a cooking competition for a second.
A judge never crowns a dish “the best” because one chef confidently says so. Every chef believes their own cooking is great. That’s human nature. Instead, the judge does something smarter. They plate several dishes. They taste each one against the same criteria. Only then do they pick a winner.
Model evaluation works exactly the same way. Here’s how the analogy breaks down into three simple rules:
First, cook several dishes. Train more than one candidate model. Never trust the first one you fit, no matter how good it looks.
Second, taste with the same spoon. Judge every model on the same held-out test set. Never evaluate on training data. That’s like a judge tasting their own cooking and declaring themselves the winner.
Third, score, don’t guess. Use the metric you chose back in EP105. Don’t switch to whichever number happens to look the most flattering after the fact.
Once you internalize this analogy, the rest of this episode becomes intuitive.
Naming the Idea: What Modeling & Evaluation Really Means
So what exactly is “modeling and evaluation”? Here’s a clean definition.
Modeling and evaluation means training multiple candidate models on the same training data. Then, you score each one on unseen test data. Crucially, you use the metric that matches your business goal, not the metric that flatters the model the most.
Picture this as a simple chain:
Candidate Models → Same Test Set → Chosen Metric → Winning Model
For example, imagine you’re predicting customer churn. You train three different models: a logistic regression, a random forest, and a gradient boosting model. Then, you score each one by recall on the test set. Why recall specifically? Because missing a customer who’s about to churn costs far more than a false alarm. That’s a real business decision, not a technical afterthought.
This is the heart of good machine learning model evaluation: the metric always follows the business goal, never the other way around.
Two Halves: Modeling and Evaluation Are Separate Steps
Here’s something first-timers often miss. “Modeling and evaluation” isn’t one task. It’s actually two distinct steps, and they happen strictly in order.
First comes modeling. This means fitting one or more algorithms to your training data. Each algorithm learns its own patterns from that data. For instance, you might fit a logistic regression and a random forest side by side, both on identical training data.
Then comes evaluation. This means scoring every one of those trained models on the test set. You use the metric you fixed back in EP105, whether that’s recall, precision, F1, or ROC-AUC.
Why does the order matter so much? Simple. You can’t judge something that doesn’t exist yet. Train first. Evaluate second. Skip this order, and you’re just guessing again.
How It Works: Five Steps to a Trustworthy Score
Let’s break the entire process into five concrete steps. Follow these in sequence, and you’ll never wonder whether your model comparison was fair.
Step 1: Train a baseline. Fit the simplest reasonable model first. This might feel unnecessary, but it isn’t. Your baseline sets the bar that everything else must beat. Without one, you have no reference point.
Step 2: Train candidates. Fit one or two more capable models on the exact same training data. Consistency here matters enormously. If your baseline and your candidates see different data, your comparison means nothing.
Step 3: Predict on test data. Generate predictions only on data no model has seen during training. This is non-negotiable. Test data must stay untouched until this exact moment.
Step 4: Score with the chosen metric. Compute the EP105 metric for every model, not just accuracy. Accuracy alone often hides more than it reveals, especially on imbalanced datasets.
Step 5: Compare and choose. Pick the model that wins on the metric that matters. Then, write down why. This last part matters more than people realize. A documented reason protects you later, when someone asks why you chose this model over that one.
Interestingly, evaluation itself becomes a loop. You train, you score, you compare. Then you repeat until a model clearly beats the baseline on your metric. How long does this take? It depends. A quick comparison on a small dataset might take a few minutes. A well-scoped capstone with three or four candidates typically takes about an hour. Meanwhile, tuning hyperparameters before your final comparison can stretch into a day or more, and that’s completely normal in real projects.
Why It Matters: Accuracy vs. the Right Metric
Let’s compare two approaches side by side. This contrast makes the stakes crystal clear.
Without evaluation discipline, choosing a model means picking whichever one has the highest accuracy. Full stop. Success gets judged once, on whatever’s convenient at the time. On imbalanced classes, a model that always predicts “no churn” can look 95% accurate while being completely useless in practice. And overfitting? It only gets discovered after the model fails quietly in production, usually at the worst possible time.
With evaluation discipline, everything changes. The model that wins is the one that wins on the metric you fixed in EP105, not just accuracy. You compare train and test scores together, not just one flattering snapshot. Precision, recall, and F1 expose what raw accuracy conveniently hides. Overfitting gets caught the moment train and test scores start pulling apart, long before it reaches production.
The difference isn’t subtle. It’s the difference between a model that works and a model that only appears to work.
The Toolkit: Models and Metrics Worth Knowing
You don’t need dozens of algorithms to get started. In fact, a small toolkit covers most real-world projects.
Logistic Regression makes an excellent baseline. It’s fast, explainable, and works well across most classification problems. Always start here before reaching for anything fancier.
Random Forest combines many decision trees that vote together. This makes it sturdy and genuinely hard to fool with noisy data.
Gradient Boosting builds trees one after another. Each new tree specifically fixes the mistakes made by the previous one. This often squeezes out extra performance, though it takes more tuning.
On the metrics side, you’ll lean on a handful of core concepts. Accuracy tells you how often the model is right overall. Precision tells you how often its “positive” predictions are actually correct. Recall tells you how many real positives the model actually catches. F1 balances precision and recall into a single number. And underneath all of these sits the confusion matrix, the four-way breakdown that every other metric is built from.
Master this toolkit, and you’ll handle the vast majority of classification problems you encounter.
Three Checks Before Any Score Earns Your Trust
Before you trust any score, run these three checks. Skipping them is how confident beginners get burned.
First, check the train versus test gap. If a model scores near-perfect on training data but performs poorly on test data, it has memorized the data, not learned from it. This is the single most common red flag in beginner projects.
Second, examine the confusion matrix in detail. Here’s the math, kept simple. Precision equals correct positive calls divided by all positive calls the model made. Recall equals correct positive calls divided by all the real positives that actually existed. These two numbers, together, tell a far richer story than accuracy alone ever could.
Third, use cross-validation. This reruns your train-test split several times. Why bother? Because one lucky test set shouldn’t get to decide the winner all by itself. Cross-validation smooths out that luck factor.
Here’s a simple rule of thumb worth memorizing: if your train accuracy sits far above your test accuracy, your model has memorized, not learned.
Hands-On: Training a Baseline and a Candidate
Theory only takes you so far. Let’s write actual code.
First, using the exact train-test split from EP105, we train a baseline and a candidate model. Both see identical training data, which keeps the comparison fair.
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
# same training data for both models
baseline = LogisticRegression(max_iter=1000)
baseline.fit(X_train, y_train)
candidate = RandomForestClassifier(random_state=42)
candidate.fit(X_train, y_train)
print("Both models trained on the same data")
Notice what’s happening here. We import Logistic Regression and Random Forest Classifier from scikit-learn. Then, we fit our baseline on X_train and y_train. Next, we fit our candidate on that same exact data. Because both models train on identical data, any difference in their later scores comes purely from the algorithm itself, not from a data mismatch. That’s the entire point of a fair comparison.
Hands-On: Scoring Both Models Fairly
Now comes the moment of truth. Let’s score both models using the metric we committed to back in EP105.
from sklearn.metrics import classification_report, recall_score
models = {"Baseline": baseline, "Candidate": candidate}
for name, model in models.items():
preds = model.predict(X_test)
# the metric we committed to back in EP105
recall = recall_score(y_test, preds)
print(name, "recall:", round(recall, 3))
print(classification_report(y_test, preds))
Here’s what this code does, step by step. We import classification_report and recall_score from scikit-learn’s metrics module. Then, we place both trained models into a dictionary, so we can loop through them cleanly instead of repeating code. For each model, we generate predictions on X_test, which is data the model has genuinely never seen before. We compute recall_score because that’s the metric we committed to earlier. Finally, classification_report hands us precision, recall, and F1 in a single call. No manual math required. This is where you actually compare your baseline against your candidate, side by side, on the metric that truly matters for your project.
Three Mistakes First-Timers Make
Even with the right process, mistakes happen. Watch out for these three traps specifically.
Mistake one: scoring on training data. This means reporting how well a model performs on data it already memorized. It feels reassuring, but it tells you nothing about real-world performance.
Mistake two: trusting accuracy alone. A model that always predicts the majority class can still score deceptively high. On imbalanced datasets, this trap catches almost everyone at least once.
Mistake three: skipping the confusion matrix. Never checking which kind of mistake your model actually makes is a costly oversight. False positives and false negatives carry very different costs, depending on your project. Ignoring that distinction means ignoring half the story.
Avoid these three traps, and you’ll already be ahead of most beginners.
Quick Recap
Let’s tie everything together.
Modeling and evaluation are two separate steps. First, you train several candidates. Then, you score them the same honest way. The cooking-competition mindset captures this perfectly: cook more than one dish, and taste every one with the same spoon.
Accuracy alone can lie to you. Precision, recall, and F1 all come from the confusion matrix, and together they tell the fuller story. A model that looks great on training data but performs weakly on test data has memorized, not learned. And ultimately, the winning model is simply the one that beats the baseline on the metric you chose back in EP105.
That’s machine learning model evaluation in a nutshell: honest comparisons, consistent metrics, and healthy skepticism toward any number that looks too good to be true.
What’s Next: Deployment & Presentation
You’ve trained your models. You’ve scored them honestly. You’ve picked a winner.
So, what happens next? In EP107, we take that winning model out of the notebook entirely. We’ll cover packaging it, deploying it, and presenting the results to people who don’t know what a confusion matrix is, and honestly, don’t need to. That’s CAPSTONE Part 3: Deployment & Presentation, and it completes the journey from raw idea to something real people can actually use.
If this article helped clarify things, the full video walks through every example visually, including the live code demos. Watch EP106: CAPSTONE Part 2 — Modeling & Evaluation on the Intelevo YouTube channel, and subscribe so you don’t miss EP107.