Beginner

14 min read

#Regression#Scikit-Learn#Python#Hands-On#Joblib

Hands-On Regression with Scikit-Learn

Step-by-step practical guide: synthetic dataset generation, train/test splitting, StandardScaler normalization, LinearRegression, metrics (MAE, RMSE, R²), and model serialization with Joblib.

Hands-On Regression with Scikit-Learn

1. Setup & Environment Imports#

To build and evaluate a reproducible regression workflow, we import numpy and pandas for numerical operations, scikit-learn for data generation, model fitting, scaling, and evaluation metrics, and joblib for model artifact persistence.


🐍 PythonInteractive WebAssembly
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Scikit-Learn modules
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Serialization
import joblib

# Plot styling
%matplotlib inline
plt.style.use('seaborn-v0_8')

2. Dataset Synthesis & Statistical Inspection#

We generate a synthetic multivariate regression dataset mimicking continuous property valuation. Generating controlled synthetic data ensures full reproducibility without relying on external file downloads.


🐍 PythonInteractive WebAssembly
# Generate synthetic regression dataset with controlled noise
X, y = make_regression(
 n_samples=1000,
 n_features=3,
 noise=10.0,
 random_state=42
)

# Structure into a clean Pandas DataFrame
feature_names = ['Square_Footage_Index', 'Room_Count', 'Property_Age_Years']
df = pd.DataFrame(X, columns=feature_names)
df['Target_Valuation'] = y

print("First 5 rows of dataset:")
print(df.head())

print("\nSummary Statistics:")
print(df.describe())

3. Train-Test Splitting (Generalization Safeguard)#

To evaluate generalization capability on unseen samples, we partition the dataset into independent training (80%) and testing (20%) splits.

Never evaluate model performance on training data. High training accuracy often masks severe overfitting and provides zero guarantee of real-world generalization.


🐍 PythonInteractive WebAssembly
# Partition into train (80%) and test (20%) splits
X_train, X_test, y_train, y_test = train_test_split(
 X, y,
 test_size=0.2,
 random_state=42
)

print(f"Training split shape: {X_train.shape} features, {y_train.shape[0]} target values")
print(f"Testing split shape: {X_test.shape} features, {y_test.shape[0]} target values")

4. Feature Standardization & Leakage Prevention#

Standardizing features to zero mean ( $\mu = 0$ ) and unit variance ( $\sigma = 1$ ) ensures numerical stability.

Data Leakage Prevention: Always compute scaler statistics (fit_transform) exclusively on the training set, then apply those frozen statistics (transform) onto the test and production sets.


🐍 PythonInteractive WebAssembly
scaler = StandardScaler()

# 1. Fit and transform exclusively on training data
X_train_scaled = scaler.fit_transform(X_train)

# 2. Apply training parameters to unseen test data
X_test_scaled = scaler.transform(X_test)

print("Standardization complete. Feature mean:", np.mean(X_train_scaled, axis=0).round(4))
print("Feature variance:", np.var(X_train_scaled, axis=0).round(4))

5. Model Fitting & Coefficient Interpretation#

We initialize Ordinary Least Squares LinearRegression and optimize weights using the closed-form Normal Equation $(\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$ .


🐍 PythonInteractive WebAssembly
model = LinearRegression()

# Train the model
model.fit(X_train_scaled, y_train)

print("Model training completed successfully.")
print(f"Learned Coefficients (Weights): {model.coef_}")
print(f"Learned Intercept (Bias): {model.intercept_:.2f}")

# Interpret feature impact
for name, coef in zip(feature_names, model.coef_):
 print(f"• {name}: {coef:+.2f} per 1.0 standard deviation change")

6. Out-of-Sample Predictions#

Generate predictions on the held-out test set using the scaled feature vectors.


🐍 PythonInteractive WebAssembly
# Predict continuous targets
y_pred = model.predict(X_test_scaled)

# Comparison table (first 5 samples)
comparison_df = pd.DataFrame({
 'Actual Valuation': y_test[:5],
 'Predicted Valuation': y_pred[:5],
 'Absolute Error': np.abs(y_test[:5] - y_pred[:5])
})

print("\nActual vs. Predicted Sample Comparison:")
print(comparison_df)

7. Model Evaluation Metrics (MAE, MSE, RMSE, R²)#

We compute quantitative regression metrics across the test distribution to validate model performance:


🐍 PythonInteractive WebAssembly
mae = mean_absolute_error(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_test, y_pred)

print("-" * 50)
print(f"Mean Absolute Error (MAE): {mae:.2f}")
print(f"Mean Squared Error (MSE): {mse:.2f}")
print(f"Root Mean Squared Error (RMSE): {rmse:.2f}")
print(f"Coefficient of Determination (R²): {r2:.4f}")
print("-" * 50)

if r2 > 0.8:
 print("Excellent fit: The model explains >80% of variance in test data.")
elif r2 > 0.5:
 print("Moderate fit: Baseline captured, hyperparameter or feature engineering recommended.")
else:
 print("Suboptimal fit: Underfitting detected.")

8. Model Artifact Persistence with Joblib#

Save both the fitted model estimator and the fitted preprocessor pipeline to disk for subsequent production microservice deployment without retraining.


🐍 PythonInteractive WebAssembly
# Persist estimator and scaler
joblib.dump(model, 'regression_model.pkl')
joblib.dump(scaler, 'regression_scaler.pkl')

print("Serialized artifacts saved: regression_model.pkl and regression_scaler.pkl")

9. Production Inference Simulation#

Simulate receiving a new, raw real-world data point, preprocessing it through the loaded scaler, and generating an instant prediction.


🐍 PythonInteractive WebAssembly
# Load persisted artifacts
loaded_model = joblib.load('regression_model.pkl')
loaded_scaler = joblib.load('regression_scaler.pkl')

# New unseen property listing [Square_Footage_Index, Room_Count, Property_Age_Years]
new_sample = np.array([[5.0, 3.0, 10.0]])

# Apply loaded scaler
new_sample_scaled = loaded_scaler.transform(new_sample)

# Generate prediction
predicted_val = loaded_model.predict(new_sample_scaled)[0]

print(f"Input Features: {new_sample[0]}")
print(f"Estimated Valuation: ${predicted_val:,.2f}")

10. Residual & Parity Visualization#

A parity plot compares observed vs. predicted targets. Data points aligning closely along the identity line ( $y = x$ ) represent high model fidelity.


🐍 PythonInteractive WebAssembly
plt.figure(figsize=(9, 5))
plt.scatter(y_test, y_pred, alpha=0.6, edgecolors='none', color='#2563eb', s=40, label='Test Observations')
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--', lw=2, label='Ideal Parity (y = x)')

plt.xlabel('Ground Truth Actual Target')
plt.ylabel('Model Predicted Target')
plt.title('Regression Parity Plot: Actual vs. Predicted')
plt.legend(loc='upper left')
plt.grid(True, linestyle=':', alpha=0.6)
plt.tight_layout()
plt.show()

11. Pipeline Engineering Summary#

In this hands-on guide, we executed a complete end-to-end regression pipeline:

Data Generation: Created clean, structured synthetic data with known noise parameters.
Splitting: Protected test integrity via strict train-test separation.
Leakage Control: Scaled features using training statistics exclusively.
Estimation: Fit an interpretable OLS linear estimator.
Diagnostics: Quantified residual distributions via MAE, RMSE, and $R^2$ .
Deployment: Persisted pipeline artifacts with joblib for zero-latency inference.

Knowledge Checkpoint

Regression Models & Scikit-Learn Checkpoint

Q1.Why is Lasso regression (L1 penalty) frequently used for feature selection?

ABecause its diamond-shaped constraint region drives less important feature coefficients to exact zero.

BBecause it scales all features to $[0, 1]$ automatically.

CBecause it works only on categorical features.

DBecause it solves linear equations using Newton-Raphson.

Q2.Why is Adjusted $R^2$ preferred over standard $R^2$ when evaluating multiple linear regression models with many features?

AStandard $R^2$ artificially increases or stays constant every time a new feature is added, whereas Adjusted $R^2$ penalizes redundant non-informative predictors.

BAdjusted $R^2$ is bounded between 0 and 100, while $R^2$ is unbounded.

CStandard $R^2$ cannot handle continuous target variables.

DAdjusted $R^2$ converts regression to classification.

Q3.What metric penalizes large prediction errors more severely: Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE)?

ARMSE, because it squares individual error residuals before averaging.

BMAE, because it takes the absolute value.

CBoth penalize large errors linearly.

DNeither, R-squared is the only metric measuring error magnitude.

Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson

Mastering Supervised Machine Learning

Next Lesson

Hands-On Classification with Scikit-Learn