Feature Engineering & Imbalanced Data Strategies

Focus: Data Quality, Feature Pipelines, Categorical Encoding, and Class Imbalance Tools: Scikit-Learn (sklearn), Imbalanced-Learn (imblearn) Level: Intermediate to Advanced

---

Table of Contents#

1. Introduction: Data Quality Over Raw Algorithms
2. Feature Engineering Deep Dive

• 2.1 Synthetic Dataset Generation
• 2.2 Missing Value Imputation
• 2.3 Categorical Encoding
• 2.4 Feature Scaling & Normalization
• 2.5 Unified Preprocessing Pipeline

1. Handling Imbalanced Datasets

• 3.1 The Accuracy Paradox
• 3.2 Resampling Techniques (Undersampling vs. Oversampling)
• 3.3 SMOTE (Synthetic Minority Over-sampling Technique)
• 3.4 Cost-Sensitive Learning & Class Weights

1. End-to-End Leak-Proof Machine Learning Pipeline
2. Performance Comparison: Baseline vs. Resampled
3. Interview Preparation Cheat Sheet
4. Conclusion & Key Takeaways

---

1. Introduction: Data Quality Over Raw Algorithms#

In production machine learning workflows, input feature quality dictates the performance ceiling of any model.

Even state-of-the-art gradient boosters (XGBoost/LightGBM) or deep neural networks will produce poor results if data exhibits:

• Unhandled missing values or uncalibrated default imputations.
• High-cardinality categorical variables mapped to invalid numerical scales.
• Disparate feature scales distorting distance metrics and gradient steps.
• Severe class imbalance where positive events occur in $<1\%$ of samples.

---

2. Feature Engineering Deep Dive#

2.1 Synthetic Dataset Generation#

We simulate a real-world Bank Loan Approval task containing numerical attributes (Income, Age, Credit Score), categorical attributes (Employment Type, City), missing entries, and a skewed binary target (Approved vs. Rejected).

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix

from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline

# Reproducibility seed
np.random.seed(42)

# Generate synthetic tabular dataset
n_samples = 1000
data = pd.DataFrame({
 'Age': np.random.randint(18, 70, n_samples),
 'Income': np.random.normal(50000, 15000, n_samples),
 'Loan_Amount': np.random.normal(10000, 5000, n_samples),
 'Employment_Type': np.random.choice(['Salaried', 'Self-Employed', 'Unemployed', 'Freelance'], n_samples),
 'City': np.random.choice(['New York', 'London', 'Tokyo', 'Paris', 'Berlin'], n_samples),
 'Credit_Score': np.random.randint(300, 850, n_samples)
})

# Introduce Missing Values (10% in Income, 5% in Employment_Type)
data.loc[np.random.rand(n_samples) < 0.10, 'Income'] = np.nan
data.loc[np.random.rand(n_samples) < 0.05, 'Employment_Type'] = np.nan

# Target generation with severe imbalance (approx 90% Rejected [0], 10% Approved [1])
prob = 1 / (1 + np.exp(-(data['Income'] - 40000)/10000 - (data['Credit_Score'] - 600)/50))
data['Approved'] = (np.random.rand(n_samples) < prob * 0.2).astype(int)

print("Dataset Dimensions:", data.shape)
print("\nTarget Class Distribution:")
print(data['Approved'].value_counts(normalize=True))
print("\nMissing Values Count:")
print(data.isnull().sum())

2.2 Missing Value Imputation#

Dropping rows containing missing values discards valuable statistical signal. Proper imputation strategies include:

• Numerical Features: Median imputation is preferred over mean when distributions exhibit skewness or outliers.
• Categorical Features: Most-frequent (mode) imputation or creating an explicit "Missing" category.

2.3 Categorical Encoding#

• One-Hot Encoding: Encodes nominal variables without natural ordering (e.g., City). Generates $K$ binary indicator columns. Set handle_unknown='ignore' to handle novel categories in inference data.
• Ordinal / Target Encoding: Encodes categories with meaningful hierarchy (e.g., Low $\rightarrow$ Medium $\rightarrow$ High) or uses regularized out-of-fold target statistics for high cardinality.

2.4 Feature Scaling & Normalization#

Standardization ($\mu=0, \sigma=1$) is essential for gradient-based solvers (Logistic Regression, Neural Networks) and distance-based estimators (KNN, SVM).

2.5 Unified Preprocessing Pipeline#

Using ColumnTransformer guarantees consistent transformations across train and test partitions without leakage.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Identify feature types
numeric_features = ['Age', 'Income', 'Loan_Amount', 'Credit_Score']
categorical_features = ['Employment_Type', 'City']

# Numerical Pipeline: Median Imputer -> StandardScaler
numeric_transformer = Pipeline(steps=[
 ('imputer', SimpleImputer(strategy='median')),
 ('scaler', StandardScaler())
])

# Categorical Pipeline: Mode Imputer -> OneHotEncoder
categorical_transformer = Pipeline(steps=[
 ('imputer', SimpleImputer(strategy='most_frequent')),
 ('encoder', OneHotEncoder(handle_unknown='ignore'))
])

# Master ColumnTransformer
preprocessor = ColumnTransformer(
 transformers=[
 ('num', numeric_transformer, numeric_features),
 ('cat', categorical_transformer, categorical_features)
 ]
)

print("Unified ColumnTransformer initialized.")

---

3. Handling Imbalanced Datasets#

3.1 The Accuracy Paradox#

When the positive class represents only $5\%$ of the samples, a naive classifier predicting all zeros achieves $95\%$ accuracy while detecting $0\%$ of positive instances. Models must be evaluated using Precision, Recall, F1-Score, and PR-AUC.

3.2 Resampling Techniques (Undersampling vs. Oversampling)#

3.3 SMOTE (Synthetic Minority Over-sampling Technique)#

SMOTE selects a minority instance $x_i$, finds its $k$-nearest minority neighbors, selects a random neighbor $x_{zi}$, and synthesizes a new point:

$x_{new} = x_i + \lambda \cdot (x_{zi} - x_i), \quad \lambda \sim U(0, 1)$

Data Leakage Rule: SMOTE must strictly be applied within the training folds during Cross-Validation. Never apply SMOTE to validation or test datasets.

---

4. End-to-End Leak-Proof Machine Learning Pipeline#

Standard Scikit-Learn Pipeline objects cannot handle resampling methods during cross-validation. We use imblearn.pipeline.Pipeline to guarantee that resampling occurs only on training folds.

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# 1. Split Data into Train and Test
X = data.drop('Approved', axis=1)
y = data['Approved']

X_train, X_test, y_train, y_test = train_test_split(
 X, y, test_size=0.2, random_state=42, stratify=y
)

# 2. Build Full Pipeline: Preprocessor -> SMOTE -> Classifier
full_pipeline = ImbPipeline(steps=[
 ('preprocessor', preprocessor),
 ('sampler', SMOTE(random_state=42)),
 ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

# 3. Train Pipeline (Applies SMOTE strictly to training data)
full_pipeline.fit(X_train, y_train)

# 4. Predict on Untouched Test Set
y_pred = full_pipeline.predict(X_test)

print("\nClassification Report (With Preprocessing & SMOTE):")
print(classification_report(y_test, y_pred))

# 5. Confusion Matrix Visualization
plt.figure(figsize=(6, 4))
sns.heatmap(
 confusion_matrix(y_test, y_pred),
 annot=True,
 fmt='d',
 cmap='Blues',
 xticklabels=['Pred Reject', 'Pred Approve'],
 yticklabels=['Act Reject', 'Act Approve']
)
plt.title('Confusion Matrix (SMOTE Pipeline)')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()

---

5. Performance Comparison: Baseline vs. Resampled#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Baseline Pipeline Without SMOTE
baseline_pipe = Pipeline(steps=[
 ('preprocessor', preprocessor),
 ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

baseline_pipe.fit(X_train, y_train)
y_pred_baseline = baseline_pipe.predict(X_test)

print("\nBaseline Classification Report (Without SMOTE):")
print(classification_report(y_test, y_pred_baseline))

The baseline model typically exhibits high Precision but significantly degraded Recall on minority class (1), missing numerous positive approvals. The SMOTE-augmented model balances minority Recall and F1-score.

---

6. Interview Preparation Cheat Sheet#

Q1: What is the risk of performing SMOTE before splitting data into train and test sets?#

Answer: Applying SMOTE before splitting introduces severe Data Leakage. Synthetic samples created from test instances will bleed into the training partition, creating artificial proximity between train and test sets and producing unrealistically high validation metrics that fail to generalize.

Q2: When would you use Class Weights instead of SMOTE?#

Answer: Cost-sensitive learning (class_weight='balanced') adjusts the loss function penalties without altering dataset size or synthesizing artificial points. It is preferred when:

• Computational resources are constrained.
• The feature space has complex noise where KNN interpolation might bridge class clusters.
• Training models natively supporting weighted loss (Logistic Regression, XGBoost scale_pos_weight, LightGBM is_unbalance).

Q3: How does ColumnTransformer prevent data leakage?#

Answer: ColumnTransformer adheres to Scikit-Learn's fit / transform paradigm. When included in a Pipeline, statistical parameters (mean, median, standard deviation, one-hot category vocabularies) are calculated strictly on training data during fit() and applied unmodified to test data via transform().

Q4: Why is Target Encoding risky, and how is it regularized?#

Answer: Target Encoding replaces categorical levels with the average target value of that category. If a category has very few samples (e.g., 1 row), it leaks the exact target value, leading to severe overfitting. It must be regularized using K-Fold target encoding, additive smoothing (m-estimate smoothing), or empirical Bayes shrinkage.

Q5: What is the difference between Nominal and Ordinal categorical features?#

Answer: Nominal variables have no intrinsic ranking or order (e.g., Country, Eye Color) and must be One-Hot encoded. Ordinal variables possess natural hierarchical order (e.g., Education Level: High School $<$ Bachelor's $<$ Master's $<$ PhD) and can be mapped directly to monotonic integers.

---

7. Conclusion & Key Takeaways#

1. Modular Transformers: Always structure data transformations inside Scikit-Learn ColumnTransformer and Pipeline objects.
2. Leak-Proof Resampling: Utilize imblearn.pipeline.Pipeline when combining resampling techniques like SMOTE with Cross-Validation.
3. Metric Alignment: For imbalanced classification, discard raw Accuracy in favor of Recall, F1-Score, and Precision-Recall AUC curves.