Machine Learning Lesson 2: Preparing Data (Missing Values and Text Columns)

📎 This article includes 3 downloadable practice files ↓

Most of the work in real machine learning is not the model; it is getting the data into shape. scikit-learn has two strict rules: no blanks and numbers only.

Missing values

Ten flats have no age and eight have no floor. You can drop those rows, but you lose data. The usual beginner fix is to fill numbers with the median: the middle value, which a single strange value cannot drag around the way it drags the average.

Text columns

City is text. A model cannot multiply “Pune” by anything. pd.get_dummies makes one column per city holding 1 or 0 (called one-hot encoding). Do not just number them 1, 2, 3, 4: the model would think Lucknow is “more” than Jaipur.

# Lesson 2: preparing data - missing values and text columns
import pandas as pd

flats = pd.read_csv("flats.csv")
print("Rows, columns:", flats.shape)
print("\nMissing values per column:")
print(flats.isna().sum())

# fill numbers with the median (not the mean: one odd value cannot drag it)
for col in ["age_years", "floor"]:
    flats[col] = flats[col].fillna(flats[col].median())

# models need numbers: turn the city text into 0/1 columns
X = pd.get_dummies(flats.drop(columns="price_lakh"), columns=["city"], dtype=int)
print("\nColumns the model will see:")
print(list(X.columns))
print("\nFirst 3 rows:")
print(X.head(3).to_string())
print("\nMissing values left:", int(X.isna().sum().sum()))

Rows, columns: (400, 6)

Missing values per column:
city           0
area_sqft      0
bedrooms       0
age_years     10
floor          8
price_lakh     0
dtype: int64

Columns the model will see:
['area_sqft', 'bedrooms', 'age_years', 'floor', 'city_Indore', 'city_Jaipur', 'city_Lucknow', 'city_Pune']

First 3 rows:
   area_sqft  bedrooms  age_years  floor  city_Indore  city_Jaipur  city_Lucknow  city_Pune
0        471         2        9.0   13.0            1            0             0          0
1        978         2       22.0    0.0            0            1             0          0
2       1352         3       24.0    7.0            0            0             1          0

Missing values left: 0

The four cities became four 0/1 columns and no blanks are left. This table is now ready for any scikit-learn model.

Practice

Download the script and data file below into one folder, open a terminal there and run python lesson-NN.py. Your numbers should match the expected output exactly (the data and random seeds are fixed). Then change one thing (a column, a setting) and see what moves.