In EP45, we built the idea of a decision tree by hand. We played a game of 20 Questions, and we met a small formula called Gini impurity. Today, that idea becomes real code.
This article is the companion guide for EP46 of the Intelevo Machine Learning Series on YouTube. Watch the video above for the full walkthrough, then use this article as your reference. Together, they cover a complete decision tree implementation in Python, from raw data to a tuned, visualized model.
Decision trees remain one of the most popular models in machine learning, and for good reason. Unlike many algorithms, a decision tree explains its own reasoning. Anyone can trace its logic, one question at a time, and understand exactly why it reached a particular answer. This transparency makes trees a favorite choice for finance, healthcare, and any field where explaining a decision matters as much as making one.
By the end of this article, you will fit a model, draw it, read it, and check whether it actually learned anything. Let’s get started.
Why Machines Need Numbers, Not Words
Our dataset predicts whether someone plays tennis. It uses four columns: Outlook, Humidity, Wind, and Play. Each one holds words, not numbers. Outlook says Sunny, Overcast, or Rain. Humidity says High or Normal.
Here’s the catch. Scikit-learn cannot read words. It only understands numbers. So, before we train anything, we need to encode every categorical column.
This step feels new, but it isn’t. We used the same trick in earlier regression episodes. Encoding simply assigns a number to each category:
import pandas as pd
df["Outlook"] = pd.factorize(df["Outlook"])[0]
df["Humidity"] = pd.factorize(df["Humidity"])[0]
df["Wind"] = pd.factorize(df["Wind"])[0]
df["Play"] = pd.factorize(df["Play"])[0]
After this step, Sunny becomes 0, Overcast becomes 1, and Rain becomes 2. High becomes 0, and Normal becomes 1. The logic stays identical. Only the labels change.
This is a small step, but skip it, and your model throws an error immediately. So, always encode first.
Consider a single row from our dataset. Before encoding, it might read: Sunny, High, Weak, No. After encoding, that same row becomes: 0, 0, 0, 0. The meaning stays exactly the same. Only the format changes. A decision tree still asks, “Is Outlook Sunny?” It just checks the condition using a number instead of a word. This distinction feels subtle, but it unlocks the entire rest of the workflow.
Building Your First Decision Tree
With clean, numeric data, we can finally build the model. Import DecisionTreeClassifier from scikit-learn’s tree module, then create an instance with three settings:
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(
criterion="gini",
max_depth=3, # guardrail against overfitting
random_state=42 # same tree, every run
)
model.fit(X_train, y_train)
Each parameter has a job. First, criterion="gini" tells the model to use the impurity measure from EP45. Next, max_depth=3 limits how many questions the tree can ask. This keeps the model from memorizing every quirk in the data. Finally, random_state=42 locks in reproducibility, so you get the same tree every time you run this code.
Then comes the real work: model.fit(X_train, y_train). This single line does everything we discussed in theory. It searches every column, tests every possible split, and picks whichever question separates the data best. It repeats this process at every new pile, until it hits the depth limit.
That’s it. Your first trained decision tree now exists in memory.
Seeing the Tree You Built
Training a model is useful, but seeing it is even better. Scikit-learn includes a built-in function for this: plot_tree.
from sklearn.tree import plot_tree
import matplotlib.pyplot as plt
plot_tree(
model,
feature_names=cols,
class_names=["No", "Yes"],
filled=True,
rounded=True
)
plt.show()
This function draws every node, branch, and leaf in your tree. Two options make the output easier to read. First, filled=True colors each box based on its majority class. Second, rounded=True softens the corners, so the diagram looks cleaner.
Once you run this, you’ll see something like this: the root node asks, “Is Outlook Overcast?” If yes, the tree already knows the answer. It’s a pure leaf, and it always predicts Yes. If no, the tree asks a follow-up question about Humidity, and splits again into two more pure leaves.
Every single box in that diagram represents a question your model chose on its own. Nobody told it to check Outlook first. It figured that out by testing every option and picking the one that worked best.
How to Read Any Decision Tree Node
Once you understand one node, you can read any decision tree, from any dataset. Every box gives you exactly four pieces of information.
First, there’s the question itself, like “Outlook = Overcast?” This condition decides whether a row moves left or right.
Second, there’s gini, which measures how mixed the pile still is. A gini of zero means the pile holds only one class. A higher gini means the pile stays jumbled.
Third, there’s samples. This number tells you how many training rows landed in this exact node.
Fourth, there’s value, written as something like [4, 4]. These are the class counts inside the pile. Whichever number is larger becomes the node’s predicted answer.
So, next time you see a strange-looking tree diagram, don’t panic. Just read the four numbers, one node at a time, and the whole picture comes together quickly.
Let’s walk through one real example. Suppose a node reads: “Outlook = Overcast?”, gini = 0.459, samples = 8, value = [4, 4]. This tells a clear story. Eight rows reached this node. Four predicted No, and four predicted Yes, so the pile stays perfectly mixed. That’s why gini sits near its maximum. After the split, one branch becomes pure, with a gini of 0.0, while the other branch still needs another question. This is exactly how a tree narrows down uncertainty, one node at a time, until every leaf reaches a confident answer.
Which Feature Matters Most?
Trees don’t just make predictions. They also keep score of their own decisions. After training, call this line:
importances = model.feature_importances_
# array([0.52, 0.31, 0.17])
This array returns one score per feature. In our example, Outlook scores 0.52, Humidity scores 0.31, and Wind scores 0.17. Higher numbers mean the feature contributed more to reducing impurity across the tree.
Notice something familiar here. Outlook scores highest, and that confirms exactly what we saw back in EP45. The tree chose Outlook first because it splits the data best. Now, we have hard numbers to back up that observation.
This makes feature importance a powerful diagnostic tool. If a feature you expected to matter scores near zero, that’s worth investigating further.
max_depth: The Single Dial That Changes Everything
Among all the parameters in DecisionTreeClassifier, one stands above the rest: max_depth. This single number completely reshapes your tree’s personality.
Set max_depth to 1, and your tree only asks one question before guessing. This tree is too simple. It misses real patterns hiding in the data, and its predictions stay shallow and generic.
Leave max_depth as None, with no limit, and something different happens. The tree keeps splitting until every training row gets its own leaf. At first, this sounds impressive. However, it isn’t learning anymore. It’s memorizing, and memorized answers don’t generalize to new data.
Somewhere between these extremes sits the sweet spot. A value like max_depth=3 usually works well for small datasets. It gives the tree enough room to learn the underlying pattern, without chasing every random quirk.
So, treat max_depth like a dial, not a fixed setting. Try a few values, compare results, and pick whichever balances accuracy with simplicity.
A Text-Only Alternative: export_text
Sometimes, a full diagram feels like overkill. Maybe you’re working in a terminal, or logging results to a file. For these situations, scikit-learn offers a lighter option: export_text.
from sklearn.tree import export_text
print(export_text(model, feature_names=cols))
This prints the exact same tree as before, but as plain, indented text. Each branch gets marked with pipe and dash symbols, like this:
|--- Outlook <= 0.50
| |--- class: Yes
|--- Outlook > 0.50
| |--- Humidity <= 0.50
| | |--- class: No
| |--- Humidity > 0.50
| | |--- class: Yes
This format works well for quick checks, automated logs, or situations where rendering an image isn’t practical. It carries the same information as plot_tree, just without the visuals.
Did Your Tree Learn or Memorize?
Here’s the question that matters most: did your tree actually learn something useful, or did it just memorize the training data? One metric answers this clearly: accuracy.
from sklearn.metrics import accuracy_score
accuracy_score(y_test, model.predict(X_test))
This function compares your model’s predictions against the true answers. Run it twice, once on training data and once on test data, and compare the results.
In our example, train accuracy comes out to 92%. However, test accuracy, on data the model has never seen, drops to 78%. This gap matters. A big difference between train and test accuracy is the clearest sign of overfitting.
The formula behind accuracy stays refreshingly simple:
Accuracy = Correct guesses / Total guesses
So, always check both numbers. Train accuracy alone tells you nothing about how your model performs in the real world.
Five Habits for Reliable Decision Trees
Before you ship any decision tree model, build these five habits into your workflow.
First, encode every categorical column into numbers before fitting. Scikit-learn cannot read text, no matter how clean it looks.
Second, always set random_state. This keeps your results reproducible, so your tree looks the same on every run.
Third, split your data into train and test sets before fitting anything. Never let your model peek at test data early. That single mistake invalidates your entire evaluation.
Fourth, skip feature scaling entirely. Unlike Ridge or Lasso regression, decision trees don’t care about the scale of your numbers. Splits work the same whether your values range from 0 to 1 or 0 to 1,000.
Fifth, compare train and test accuracy every single time. This habit catches overfitting early, before it becomes a bigger problem down the line.
The Complete Pipeline
Let’s bring everything together into one script. This covers the full journey, from raw data to a trained, evaluated decision tree.
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Test accuracy:", accuracy_score(y_test, predictions))
Twelve lines. That’s all it takes to encode, split, train, predict, and evaluate a working decision tree. Every concept from EP45 lives inside this short script.
Real-World Uses of Decision Trees
Before we wrap up, it helps to see where this model shows up outside a tutorial. Banks use decision trees to approve or reject loan applications, since regulators often require a clear explanation for every decision. Hospitals use them to flag patients at risk of specific conditions, because doctors need to trust and verify the reasoning behind each alert. E-commerce platforms use them for recommendation systems, and manufacturers use them to catch defective products on an assembly line.
In every one of these cases, the same advantage applies. A decision tree explains itself. Nobody has to guess why it made a particular call. That transparency, combined with the simple implementation you just learned, is exactly why decision trees remain a staple in any practical machine learning toolkit.
Full Circle: If You Can Read a Flowchart, You Can Read This
Let’s recap the four ideas that matter most. First, encode, then fit. Three scikit-learn calls handle all the work that the 20 Questions game represented back in EP45. Second, plot_tree turns that logic into a single picture anyone can follow. Third, feature_importances_ keeps score, showing exactly which question mattered most. Fourth, max_depth acts as your dial, sitting somewhere between too simple and memorizing every quirk.
If you can read a flowchart, you can read a decision tree. This time, you didn’t just learn the theory. You built one, visualized one, and tuned one yourself.
Frequently Asked Questions
Do I need to scale my features before training a decision tree? No, you don’t. Decision trees split data based on thresholds, not distances. So, unlike Ridge or Lasso regression, scaling adds no benefit here.
Why does my decision tree implementation in Python throw an error on string columns? Scikit-learn’s DecisionTreeClassifier only accepts numeric input. If your dataset still contains words, encode every categorical column first, using pandas’ factorize function or LabelEncoder.
What happens if I don’t set max_depth? Without a limit, your tree keeps splitting until every leaf becomes pure. This usually leads to overfitting, since the model starts memorizing instead of learning general patterns.
How do I know if my model is overfitting? Compare train accuracy against test accuracy. A small gap is normal. However, a large gap, like 92% versus 78%, signals that your tree memorized the training data instead of learning from it.
What’s Next: Naive Bayes Classifier
Next week, in EP47, we meet a completely different kind of thinker: the Naive Bayes Classifier. Unlike decision trees, this model reasons entirely in probabilities, not questions. If you enjoyed learning how trees split data, you’ll love seeing how probability-based models approach the exact same problem from a totally different angle.
Watch the full walkthrough in the EP46 video above, then try the code yourself with your own dataset. If any step feels unclear, drop a comment below the video. I read every single one, and I’m always happy to help you get unstuck.
For more tutorials like this one, explore the rest of the Intelevo Machine Learning Series on YouTube, and check back on intuitivetutorial.com for new articles every week.