Machine Learning Lesson 7: Pipelines, Cross-Validation and Avoiding Leakage

📎 This article includes 3 downloadable practice files ↓

In Lessons 2 and 3 we filled gaps using the median of all flats, then split. That quietly lets the test rows influence training, which is called leakage, and it makes scores look better than they will be in real use.

A pipeline fixes it: filling, scaling, encoding and the model become one object. Whenever it is trained, it learns the medians from the training rows only.

Cross-validation fixes a second problem: one random split can be lucky. 5-fold cross-validation trains and tests five times on different slices and reports every score.

# Lesson 7: pipelines and cross-validation (and avoiding leakage)
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import Ridge
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score, KFold

flats = pd.read_csv("flats.csv")          # raw, with gaps: the pipeline does the cleaning
X, y = flats.drop(columns="price_lakh"), flats["price_lakh"]
num = ["area_sqft", "bedrooms", "age_years", "floor"]

prep = ColumnTransformer([
    ("num", Pipeline([("fill", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), num),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["city"]),
])
cv = KFold(n_splits=5, shuffle=True, random_state=0)
for name, est in [("Ridge regression", Ridge(alpha=1.0)), ("Random forest", RandomForestRegressor(n_estimators=200, random_state=0))]:
    pipe = Pipeline([("prep", prep), ("model", est)])
    mae = -cross_val_score(pipe, X, y, cv=cv, scoring="neg_mean_absolute_error")
    print(f"{name:17} MAE per fold: {[float(round(m, 1)) for m in mae]}  average {mae.mean():.1f} lakh")

Ridge regression  MAE per fold: [7.2, 6.7, 7.4, 7.0, 7.4]  average 7.2 lakh
Random forest     MAE per fold: [6.6, 6.2, 6.0, 6.0, 6.2]  average 6.2 lakh

The random forest is off by about 6.2 lakh on average against 7.2 for ridge regression, and it wins on every fold, so the difference is real rather than luck. Notice the script reads the raw file with its gaps: the pipeline does all the preparation.

Practice

Download the script and data file below into one folder, open a terminal there and run python lesson-NN.py. Your numbers should match the expected output exactly (the data and random seeds are fixed). Then change one thing (a column, a setting) and see what moves.