Machine Learning Lesson 4: Classification (Which Customers Will Leave?)

📎 This article includes 3 downloadable practice files ↓

Now the target is a yes/no: did the broadband customer leave (churn)? This is classification. Logistic regression, despite the name, is the standard first model for it: it gives each customer a probability of leaving.

Why accuracy is not enough

About 36% of customers left. A model that says “nobody leaves” is right 64% of the time and completely useless. So look at:

• Precision: of the customers we flagged, how many really left. Low precision wastes the retention team’s calls.
• Recall: of the customers who left, how many we caught. Low recall means lost customers we never called.

# Lesson 4: classification - which customers will leave?
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score

df = pd.read_csv("customers.csv")
print("Churn rate:", f"{df['churned'].mean():.1%}")
X = pd.get_dummies(df.drop(columns=["customer_id", "churned"]), columns=["contract"], drop_first=True, dtype=int)
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=7, stratify=y)

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
pred = model.predict(X_test)
print(f"Accuracy:  {accuracy_score(y_test, pred):.1%}")
print(f"Precision: {precision_score(y_test, pred):.1%}  (of those we flagged, how many really left)")
print(f"Recall:    {recall_score(y_test, pred):.1%}  (of those who left, how many we caught)")
print("\nConfusion matrix (rows = actual stay/leave, columns = predicted stay/leave):")
print(confusion_matrix(y_test, pred))

# probabilities let you pick your own cut-off
prob = model.predict_proba(X_test)[:, 1]
flag = prob >= 0.35
print(f"\nWith a 35% cut-off: flagged {flag.sum()}, recall {recall_score(y_test, flag):.1%}, precision {precision_score(y_test, flag):.1%}")

Churn rate: 35.9%
Accuracy:  74.4%
Precision: 70.3%  (of those we flagged, how many really left)
Recall:    50.0%  (of those who left, how many we caught)

Confusion matrix (rows = actual stay/leave, columns = predicted stay/leave):
[[141  19]
 [ 45  45]]

With a 35% cut-off: flagged 104, recall 70.0%, precision 60.6%

Reading the confusion matrix: 141 stayers were correctly left alone, 45 leavers were caught, but 45 leavers were missed. At the default 50% cut-off the model catches half the leavers. Dropping the cut-off to 35% catches 70% of them, at the price of more false alarms. The right cut-off is a business decision: what does a wasted call cost compared with a lost customer?

Practice

Download the script and data file below into one folder, open a terminal there and run python lesson-NN.py. Your numbers should match the expected output exactly (the data and random seeds are fixed). Then change one thing (a column, a setting) and see what moves.