Machine Learning Lesson 5: Decision Trees and Random Forests

📎 This article includes 3 downloadable practice files ↓

A decision tree asks a series of yes/no questions, like a flowchart. You can print it and show it to anyone. Limit its depth and it stays simple; let it grow and it memorises the training data.

# Lesson 5: decision trees and random forests
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, recall_score

df = pd.read_csv("customers.csv")
X = pd.get_dummies(df.drop(columns=["customer_id", "churned"]), columns=["contract"], drop_first=True, dtype=int)
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=7, stratify=y)

tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
print("A tree you can read:")
print(export_text(tree, feature_names=list(X.columns)))
print(f"Tree accuracy: {accuracy_score(y_test, tree.predict(X_test)):.1%}")

deep = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print(f"Unlimited tree: train {accuracy_score(y_train, deep.predict(X_train)):.1%}, test {accuracy_score(y_test, deep.predict(X_test)):.1%}  <- overfitting")

forest = RandomForestClassifier(n_estimators=300, max_depth=6, random_state=0).fit(X_train, y_train)
fp = forest.predict(X_test)
print(f"Random forest accuracy: {accuracy_score(y_test, fp):.1%}, recall {recall_score(y_test, fp):.1%}")
imp = pd.Series(forest.feature_importances_, index=X.columns).sort_values(ascending=False)
print("\nWhat matters most:")
print(imp.round(3).to_string())

A tree you can read:
|--- tenure_months <= 45.50
|   |--- contract_Yearly <= 0.50
|   |   |--- monthly_bill <= 564.50
|   |   |   |--- class: 0
|   |   |--- monthly_bill >  564.50
|   |   |   |--- class: 1
|   |--- contract_Yearly >  0.50
|   |   |--- complaints <= 1.50
|   |   |   |--- class: 0
|   |   |--- complaints >  1.50
|   |   |   |--- class: 1
|--- tenure_months >  45.50
|   |--- complaints <= 2.50
|   |   |--- monthly_bill <= 1137.50
|   |   |   |--- class: 0
|   |   |--- monthly_bill >  1137.50
|   |   |   |--- class: 0
|   |--- complaints >  2.50
|   |   |--- monthly_bill <= 720.50
|   |   |   |--- class: 0
|   |   |--- monthly_bill >  720.50
|   |   |   |--- class: 1

Tree accuracy: 70.4%
Unlimited tree: train 100.0%, test 64.0%  <- overfitting
Random forest accuracy: 72.8%, recall 47.8%

What matters most:
tenure_months      0.389
monthly_bill       0.265
complaints         0.156
contract_Yearly    0.109
autopay            0.080

The first question the tree asks is about tenure: newer customers leave more. The unlimited tree scored 100% on training data and only 64% on the test set. That gap is overfitting, the most common mistake in machine learning.

A random forest trains 300 trees on random slices of the data and lets them vote. It is harder to read but more stable, and its feature importances tell you what drives the prediction: here tenure, then bill, then complaints. On this data it did not beat logistic regression; with simple, mostly straight-line patterns, simple models often win. Always test, never assume.

Practice

Download the script and data file below into one folder, open a terminal there and run python lesson-NN.py. Your numbers should match the expected output exactly (the data and random seeds are fixed). Then change one thing (a column, a setting) and see what moves.