Machine Learning Lesson 3: Train/Test Split and Honest Evaluation

📎 This article includes 3 downloadable practice files ↓

A model that is tested on the same rows it learned from will look brilliant and fail on new data, like a student who memorised last year’s paper. So we hide a quarter of the flats, train on the rest, and test only on the hidden ones.

Two scores for regression

• MAE (mean absolute error): on average, how far off is the prediction, in the same unit as the target (lakh). Easy to explain to a manager.
• R²: how much of the variation the model explains, from 0 (useless) to 1 (perfect).

# Lesson 3: train/test split and honest evaluation
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score

flats = pd.read_csv("flats.csv")
flats[["age_years", "floor"]] = flats[["age_years", "floor"]].fillna(flats[["age_years", "floor"]].median())
X = pd.get_dummies(flats.drop(columns="price_lakh"), columns=["city"], dtype=int)
y = flats["price_lakh"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)
print("Train rows:", len(X_train), " Test rows:", len(X_test))

model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
print(f"MAE on unseen flats: {mean_absolute_error(y_test, pred):.1f} lakh")
print(f"R2 on unseen flats:  {r2_score(y_test, pred):.3f}")

# baseline: always guess the average price. A model must beat this.
base = [y_train.mean()] * len(y_test)
print(f"Baseline MAE (always the average): {mean_absolute_error(y_test, base):.1f} lakh")

check = pd.DataFrame({"actual": y_test.values[:5], "predicted": pred[:5].round(1)})
print("\n" + check.to_string(index=False))

Train rows: 300  Test rows: 100
MAE on unseen flats: 7.1 lakh
R2 on unseen flats:  0.924
Baseline MAE (always the average): 25.6 lakh

 actual  predicted
   89.9      100.2
  107.6      108.2
   60.0       65.8
   70.0       73.5
  101.3       97.6

On flats it never saw, the model is off by about 7 lakh on average, against 25.6 lakh for the lazy “always guess the average” baseline. That comparison is what tells you the model is worth using.

Practice

Download the script and data file below into one folder, open a terminal there and run python lesson-NN.py. Your numbers should match the expected output exactly (the data and random seeds are fixed). Then change one thing (a column, a setting) and see what moves.