This blog post explains the critical concepts of train, test, and validation sets in machine learning, detailing their importance in model training and evaluation, and providing a step-by-step guide on how to implement these splits using a California housing dataset.
Machine learning is a powerful tool that requires careful handling of data to ensure models perform well. One of the fundamental concepts in machine learning is the division of data into train, test, and validation sets. This blog post will explore these concepts in detail, using a California housing dataset as an example.
To train a machine learning model effectively, it is essential to partition the dataset into different subsets. The primary goal is to train the model on one part of the data and evaluate its performance on another. This helps in understanding how well the model generalizes to unseen data.
Initially, we can start with two splits:
For our example, we will use the California housing dataset, which contains various features such as longitude, latitude, housing median age, total rooms, and the target variable, median house value.
To begin, we load the training and testing datasets using the following code:
import pandas as pd
df_train = pd.read_csv('sample_data/california_housing_train.csv')
df_test = pd.read_csv('sample_data/california_housing_test.csv')
This gives us access to the first few rows of both datasets, allowing us to understand the structure and features available for modeling.
For our modeling purposes, we will focus on two features: housing median age and population, to predict the median house value. This is a regression problem where we aim to predict a continuous value.
We create a new DataFrame that includes only the relevant columns:
df_train_subset = df_train[['housing_median_age', 'population', 'median_house_value']]
df_test_subset = df_test[['housing_median_age', 'population', 'median_house_value']]
Next, we convert these DataFrames into NumPy arrays for easier manipulation:
X_train = df_train_subset[['housing_median_age', 'population']].to_numpy()
Y_train = df_train_subset['median_house_value'].to_numpy()
X_test = df_test_subset[['housing_median_age', 'population']].to_numpy()
Y_test = df_test_subset['median_house_value'].to_numpy()
We will use a linear regression model from the sklearn library to fit our training data:
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
model = LinearRegression()
model.fit(X_train, Y_train)
After fitting the model, we can make predictions on the training set and calculate the mean absolute error (MAE):
train_predictions = model.predict(X_train)
train_error = mean_absolute_error(Y_train, train_predictions)
To evaluate how well our model performs on unseen data, we apply it to the test set:
test_predictions = model.predict(X_test)
test_error = mean_absolute_error(Y_test, test_predictions)
It is crucial to note that the error on the test set is what we care about most, as it reflects the model's performance on new data.
While the train-test split is a good start, it is often beneficial to introduce a third split: the validation set. This allows for better tuning of the model without overfitting to the test set.
The validation set helps in assessing the model's performance during the training phase. By evaluating the model on this separate set, we can adjust parameters and improve the model before final testing.
To create a validation set, we can split our test data further into validation and holdout sets. This can be done using the train_test_split function from sklearn:
from sklearn.model_selection import train_test_split
X_val, X_hold, Y_val, Y_hold = train_test_split(X_test, Y_test, test_size=0.5)
Now we have three sets:
After training and validating our model, we can make predictions on the holdout set to get an unbiased estimate of its performance:
holdout_predictions = model.predict(X_hold)
holdout_error = mean_absolute_error(Y_hold, holdout_predictions)
This final error gives us a realistic expectation of how the model will perform in production.
In summary, understanding and implementing train, test, and validation sets is crucial for building effective machine learning models. By properly partitioning data, we can ensure that our models generalize well to unseen data, ultimately leading to better performance in real-world applications. As you continue your journey in machine learning, remember the importance of these data splits and consider exploring advanced techniques like cross-validation for even better model evaluation.
