Chapter 6: Model Evaluation & Selection

⚖️ After building a model, you must evaluate how well it performs and select the best one among alternatives.

---

🔍 Why Is This Chapter Important?

• Avoid overfitting (too good on training, bad on test).

• Detect underfitting (poor on both training and test).

• Choose the right model for your data.

• Improve model through tuning and validation.

---

📌 Key Concepts Covered:

1. Train/Test Split

2. Cross-Validation

3. Bias-Variance Trade-off

4. Overfitting vs. Underfitting

5. Hyperparameter Tuning

6. Model Selection Techniques

---

🔹 1. Train-Test Split

Divide your dataset into two parts:

• Training Set (80%) – to train the model

• Test Set (20%) – to test how well the model generalizes

python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

---

🔹 2. Cross-Validation

K-Fold Cross Validation splits data into k parts, trains on k-1 parts, tests on the remaining, and repeats.

🔁 Example: 5-Fold CV

• Divide data into 5 parts.

• Train on 4, test on 1 — repeat 5 times.

python
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5)
print(scores.mean())

📌 Benefit: More reliable than a single train/test split.

---

⚖️ 3. Bias-Variance Trade-off

Term	Meaning	Problem
High Bias	Model too simple	Underfitting
High Variance	Model too complex	Overfitting

🧠 Goal: Balance bias and variance to get best generalization.

---

🧱 4. Overfitting vs Underfitting

Type	Training Accuracy	Test Accuracy	Looks Like
Overfitting	High	Low	Memorized data
Underfitting	Low	Low	Didn’t learn enough

📌 Fix Overfitting:

• More data

• Simpler model

• Regularization

📌 Fix Underfitting:

• More features

• Complex model

• Train longer

---

🧪 5. Evaluation Metrics Recap

For Classification:

• Accuracy

• Precision / Recall / F1 Score

• Confusion Matrix

• ROC-AUC Curve

For Regression:

• MSE (Mean Squared Error)

• RMSE (Root Mean Squared Error)

• MAE (Mean Absolute Error)

• R² Score (Coefficient of Determination)

---

🔧 6. Hyperparameter Tuning

Hyperparameters are settings you configure before training the model (like number of trees, learning rate, k in KNN).

Techniques:

✅ Grid Search

Try all combinations of given parameters.

python
from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(model, param_grid, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)

✅ Randomized Search

Randomly selects parameter combinations — faster for large search space.

---

📌 Example Workflow:

1. Train/test split → baseline model

2. K-Fold Cross-Validation → better estimate

3. Choose best evaluation metric (accuracy or RMSE)

4. Tune hyperparameters with GridSearch

5. Select final model → deploy

---

💻 Hands-On Mini Project Ideas

✅ Compare Models:

• Train Logistic Regression, KNN, and SVM on the same dataset

• Use Cross-Validation to compare average accuracy

✅ Tune Model:

• Use GridSearch to tune K in KNN or max_depth in Decision Trees

---

🧠 Summary of Chapter 6

Concept	Role
Train-Test Split	Initial testing of model
Cross-Validation	Robust performance checking
Overfitting	Model too specific
Underfitting	Model too general
Hyperparameter Tuning	Improves model performance
Model Metrics	Helps compare models

---

✅ Mini Assignment:

1. Use 5-Fold Cross-Validation on a classifier and report average accuracy.

2. Build a Decision Tree and tune max_depth using GridSearch.

3. Explain a case where your model was overfitting. How did you fix it?