Video companion: This article accompanies Episode 62 of the Intelevo Machine Learning series on YouTube. Watch the full video first for the visual walkthrough, then use this article to review the concepts, revisit the code, and take notes at your own pace.
A single decision tree feels trustworthy. It draws a clean path through your data, and it gives you a clear answer. But that trust comes with a catch. Change the training data just a little, and the tree can change its mind completely. That instability has a name: high variance. And it is exactly the problem that Bagging and Random Forests solve.
In this article, we will build the idea from the ground up. We will start with a simple analogy, move to the actual mechanics, and finish with a working Python example. By the end, you will see why Random Forests remain one of the most dependable algorithms in machine learning, even years after fancier models arrived on the scene.
The Problem With Trusting One Tree
A decision tree learns by asking a series of yes-or-no questions. It splits your data at each question, narrows down the possibilities, and eventually lands on a prediction. This process is fast, and it is easy to explain to anyone, even someone with no background in statistics.
However, a single tree pays a price for that simplicity. It tends to memorize the training data too closely. As a result, it captures noise along with the real pattern. Give it a slightly different dataset, and it often produces a very different tree.
Think of a single tree as a brilliant but opinionated expert. Ask this expert once, and you get a confident answer. Ask again with slightly different information, and the expert might change their mind entirely. That inconsistency makes a single tree risky to rely on in high-stakes decisions.
So, how do we fix this? We stop relying on one opinion. Instead, we ask many.
The Big Idea: Ask the Audience
Picture a television quiz show. A contestant gets stuck on a hard question, and they reach for a lifeline: Ask the Audience. Now, they have two choices.
First, they could ask one friend. This approach is fast, but it carries risk. If that one friend happens to be biased or simply wrong, the contestant walks away with a confident, incorrect answer.
Second, they could ask the entire audience. Every person in the room casts a vote. Individual mistakes cancel each other out, and the majority answer turns out to be correct an astonishing amount of the time.
This single analogy captures the entire intuition behind Bagging and Random Forests. Instead of trusting one tree, we grow many trees. Then, we let them vote.
Naming the Idea: Bagging
That “ask the audience” strategy has a formal name in machine learning: Bagging. The term stands for Bootstrap Aggregating, and it breaks down into three clear steps.
Step 1: Bootstrap. We build many random samples from our original dataset. We pick each sample with replacement, which means the same row can appear more than once, and other rows might not appear at all.
Step 2: Train. We grow one decision tree on each of these samples. This gives us a small committee of trees, and each tree sees a slightly different version of the data.
Step 3: Aggregate. We combine every tree’s answer, either through a majority vote or through a simple average.
That’s the whole algorithm. Three steps, and no complicated math required to understand it.
Step 1 in Detail: Bootstrap Sampling
Let’s slow down and look closely at bootstrap sampling, because this step drives everything that follows.
Imagine your original dataset has five rows, labeled A through E. To create the first bootstrap sample, you randomly draw rows from this dataset, but you draw them with replacement. That means you could pick row A twice, while row B might not appear at all.
So, Sample 1 might end up as: A, A, C, D, E. Sample 2 might look like: B, B, C, D, D. Sample 3 might come out as: A, C, C, E, E.
Each sample looks a little different from the original dataset, and each sample looks different from the other samples too. Next, we train one decision tree on each of these samples.
Here is the key insight. Because each tree sees a slightly different version of reality, each tree makes slightly different mistakes. That variation is not a flaw. In fact, it is exactly what makes the final ensemble powerful.
Step 2 in Detail: Aggregating the Results
Once we have many trees, we need to combine their answers into one final decision. The method depends on the type of problem we are solving.
For classification problems, like spam detection, each tree casts a vote. Suppose five trees vote on whether an email counts as spam. Three trees say “Spam,” and two trees say “Not Spam.” Since Spam wins the majority, three votes out of five, that becomes our final answer.
For regression problems, like predicting a price, we skip the voting and average the numbers instead. Suppose five trees each predict a slightly different price for a house: Rs. 42,000, Rs. 39,500, Rs. 45,200, Rs. 41,000, and Rs. 40,800. We simply average these five numbers, which gives us Rs. 41,700 as our final prediction.
Both methods follow the same underlying logic. We combine many opinions into one confident answer.
Why Does This Actually Work?
At this point, you might wonder why averaging many noisy guesses produces a better result than trusting one careful guess. The answer comes down to how errors behave across many trees, and we can explain it without heavy math.
A single tree makes confident, and sometimes wrong, guesses. If you plotted its predictions on a target, you would see them scattered around the bullseye, sometimes close and sometimes far away. This scattering is what we call high variance.
Now, imagine growing many trees, and each one makes a different, mostly random error. When you average their predictions, the errors tend to cancel each other out. The scattered dots pull together and cluster tightly around the true answer.
This is the mathematical version of “ask the audience.” Individually, each tree stays noisy. Collectively, the group becomes remarkably accurate.
One More Twist: From Bagging to Random Forest
Bagging alone already improves accuracy, but it has one weakness. If every tree can still look at every single column in the data, the trees tend to agree on the same “star” feature. They all split on it first, and they end up looking surprisingly similar to each other. When trees look similar, they make similar mistakes, and averaging similar mistakes does not help much.
Random Forest fixes this weakness with one clever addition. In addition to bootstrapping the rows, Random Forest also randomizes the columns. At every single split, each tree can only consider a random subset of the available features.
This restriction forces the trees to disagree with each other in useful ways. One tree might split on income first, because that’s all it could see at that moment. Another tree might split on age first, for the same reason. The result is a more diverse forest of trees, and diversity translates directly into better accuracy.
How a Random Forest Gets Built, Step by Step
Let’s put the entire process together, from start to finish.
First, the algorithm bootstraps the rows. It draws a random sample, with replacement, for every single tree in the forest.
Second, it randomizes the columns. At each split point inside each tree, the algorithm only considers a random subset of the available features.
Third, it grows deep trees. Each individual tree gets to grow fully, and it’s perfectly fine if one tree slightly overfits its own bootstrap sample.
Fourth, it votes or averages. The forest combines every tree’s output into one final answer, using majority vote for classification or simple averaging for regression.
Then, the algorithm repeats this entire process for hundreds of trees. That repetition is what turns a handful of trees into a proper forest.
Key Hyperparameters to Know
Before we move to code, let’s cover three settings you will adjust most often when building a Random Forest.
n_estimators controls how many trees the forest grows. More trees generally produce steadier, more reliable answers, but the improvement slows down after a certain point. Adding a thousandth tree rarely helps as much as adding your hundredth tree did.
max_features controls how many random columns each split can consider. A smaller value forces more diversity among your trees, since each tree gets a narrower view of the data at every split.
max_depth controls how deep each individual tree can grow. Deeper trees capture more detail, but they also risk overfitting their own bootstrap sample more aggressively.
As a starting point, try somewhere between 200 and 500 trees, and leave the other settings at their defaults. This combination works well for most everyday problems, and you can fine-tune from there.
Single Tree vs. Bagging vs. Random Forest
Here is a quick comparison to lock in the differences between these three approaches.
A single tree uses all the data and all the features every time. It carries high variance, but it stays very easy to interpret, since you can literally draw the tree on paper.
Bagging adds bootstrap sampling on top of a single tree. This lowers variance considerably, but the resulting model becomes harder to interpret, since you now have many trees instead of one.
Random Forest goes one step further. It adds feature randomization to bootstrap sampling, which produces the lowest variance of the three approaches and typically the best accuracy. Like Bagging, though, it remains harder to interpret than a single tree.
Strengths and Limitations
Every technique carries trade-offs, and Random Forest is no exception.
On the strengths side, Random Forest delivers much more accuracy and stability than a single tree. It handles messy, non-linear data well, and it requires very little tuning to get solid results. It resists overfitting far better than one deep tree does, and it even calculates useful feature importance rankings as a side benefit.
On the limitations side, Random Forest loses the simple “draw the tree” interpretability that made single trees so appealing. It also trains and predicts more slowly, since you’re now running many trees instead of one. The model takes up more memory too, and it can still struggle when it needs to extrapolate far beyond the range of its training data.
Python Demo: Random Forest in Five Lines
Now, let’s see how little code this actually takes. Scikit-learn handles all the bootstrapping and voting internally, so you never write that logic yourself.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = RandomForestClassifier(
n_estimators=300,
max_features="sqrt",
random_state=42
)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))
# 0.94
Let’s walk through what each part does.
First, we import RandomForestClassifier from sklearn.ensemble, along with train_test_split from sklearn.model_selection. Then, we split our data, keeping 20 percent aside for testing so we can evaluate the model fairly.
Next, we create our model. We set n_estimators to 300, which grows 300 trees in our forest. We set max_features to "sqrt", a common default for classification problems. We also set a random_state, which keeps our results reproducible every time we run the code.
After that, we call model.fit() on our training data, and the forest learns from it. Finally, we call model.score() on our test set, which returns the model’s accuracy. In this example, our forest scores 94 percent.
Notice something important here. We never wrote a single line of resampling code, and we never wrote a single line of voting logic. Scikit-learn handles all of that internally. The API looks identical to a single DecisionTreeClassifier, but the underlying model performs considerably better.
Where You’ll See Random Forests in the Real World
Random Forests show up across many industries, precisely because they deliver strong accuracy without demanding heavy tuning.
In healthcare, teams use Random Forests to predict disease risk from patient records and lab results. The model handles messy, incomplete medical data gracefully, which makes it a practical choice for clinical settings.
In finance, Random Forests power credit scoring and fraud detection. They spot subtle patterns across large volumes of transaction data, patterns that a human analyst might easily miss.
In e-commerce, companies rely on Random Forests for product recommendations and customer churn prediction. The algorithm identifies which customers are likely to leave, so businesses can intervene before they lose them.
Bringing It All Together
Let’s recap the whole episode in a single sentence: don’t trust one opinion, ask a crowd, and let the crowd vote.
We start by bootstrapping our data into random samples. Then, we grow many trees on those samples. Random Forest adds one extra twist by randomizing the features at each split. Finally, we vote or average across every tree to arrive at one strong, confident answer.
A single tree guesses. A forest knows.
What’s Next: Boosting and AdaBoost
Random Forest builds its trees independently and combines them through a vote. But what if each tree could learn from the mistakes of the tree before it, one step at a time? That question leads us directly into Episode 63, where we explore Boosting and AdaBoost.
If this article helped clarify Bagging and Random Forests for you, please watch the full video on the Intelevo YouTube channel, and consider leaving a comment with your questions or your own use case. Your feedback genuinely shapes future episodes in this series.