Gradient Boosting

A machine learning method called Gradient Boosting creates an ensemble by combining several weak prediction models. Decision trees, which are sequentially trained to reduce errors and increase accuracy, are commonly used as these weak models. Gradient boosting can efficiently capture intricate correlations between features by combining several decision tree regressors or decision tree classifiers.

Gradient boosting's capacity to iteratively minimize the loss function is one of its main advantages. One loss function used to assess how well a machine learning model matches actual data is Mean Squared Error (MSE). MSE determines the mean of the squared discrepancies between the observed and expected values.

MAE (Mean Absolute Error) quantifies the average magnitude of errors for Gradient Boosting Regression.

Mean Absolute Error
It calculates the average discrepancy between a dataset's actual and forecasted values. Without taking direction into account, it displays the deviation between predicted and actual values.
1. Determined by utilizing absolute differences
2. Easy to calculate and understand
3. Handles every mistake equally
4. Not as susceptible to significant errors as MSE
5. Frequently employed to assess regression models

Gradient Boosting Classification

Complete Loan Default Prediction Example

10 Loans • 3 Independent Variables • 3 Decision Trees • Logistic Loss

η = 0.5Simple gradient/residual leaf valueF₀ computed from class proportion

1. Dataset

Loan	Income	Debt Ratio	Credit Score	Default Y
1	$30k	0.80	580	1
2	$35k	0.75	600	1
3	$40k	0.70	620	1
4	$45k	0.65	640	1
5	$50k	0.60	660	0
6	$55k	0.50	680	0
7	$60k	0.45	700	0
8	$65k	0.40	720	0
9	$70k	0.35	740	0
10	$80k	0.30	760	0

Y = 1 means Default; Y = 0 means No Default.

2. Calculate the Initial F₀ Correctly

There are 4 defaults among 10 loans, so the initial class probability is:

p₀ = Ŷ = 4 / 10 = 0.40

For logistic boosting, convert this probability to initial log-odds:

F₀ = ln(p₀ / (1 − p₀))

F₀ = ln(0.40 / 0.60) = ln(2/3) ≈ -0.4055

Check the conversion back to probability:

p₀ = 1 / (1 + e^−F₀) = 1 / (1 + e^0.4055) ≈ 0.4000

Important: We use this principled F₀, but from Tree 1 onward we use the requested simple leaf value γ = Σ(Y − p) / N_leaf.

3. Tree 1 — Credit Score Split

Residuals are the negative gradients for logistic loss:

rᵢ = Yᵢ − pᵢ

Initially p₀ = 0.4 for every loan.

Loans	Y	p₀	Residual Y − p₀
1–4	1	0.40	+0.60
5–10	0	0.40	−0.40

Suppose Tree 1 splits on Credit Score ≤ 650.

Credit Score ≤ 650?

YES: Loans 1–4
γ₁ = +0.6000

NO: Loans 5–10
γ₂ = −0.4000

Leaf 1

(0.6 + 0.6 + 0.6 + 0.6) / 4 = 0.6000

Leaf 2

(−0.4 × 6) / 6 = −0.4000

Apply η = 0.5

F₁ = F₀ + ηγ

Loans 1–4: F₁ = -0.4055 + (0.5)(0.6) = -0.1055

Loans 5–10: F₁ = -0.4055 + (0.5)(−0.4) = -0.6055

Group	F₁	p₁
Loans 1–4	-0.1055	0.4737
Loans 5–10	-0.6055	0.3531

4. Tree 2 — Debt Ratio Split

Now Tree 2 learns from the remaining residuals:

r₂ = Y − p₁

Group	Y	p₁	Residual
Loans 1–4	1	0.4737	+0.5263
Loans 5–10	0	0.3531	-0.3531

Suppose Tree 2 splits on Debt Ratio > 0.625.

Debt Ratio > 0.625?

YES: Loans 1–4
γ = 0.5263

NO: Loans 5–10
γ = -0.3531

Leaf 1

(4 × 0.5263) / 4 = 0.5263

Leaf 2

(6 × -0.3531) / 6 = -0.3531

Apply Tree 2

Loans 1–4: F₂ = -0.1055 + (0.5)(0.5263) = 0.1577

p₂ = 1/(1 + e^−0.1577) = 0.5393

Loans 5–10: F₂ = -0.6055 + (0.5)(-0.3531) = -0.7820

p₂ = 1/(1 + e^−(-0.7820)) = 0.3139

5. Tree 3 — Income Split

r₃ = Y − p₂

Group	Y	p₂	Residual
Loans 1–4	1	0.5393	+0.4607
Loans 5–10	0	0.3139	-0.3139

Suppose Tree 3 splits on Income ≤ $47.5k.

Income ≤ $47.5k?

YES: Loans 1–4
γ = 0.4607

NO: Loans 5–10
γ = -0.3139

Leaf 1

(4 × 0.4607) / 4 = 0.4607

Leaf 2

(6 × -0.3139) / 6 = -0.3139

Apply Tree 3

Loans 1–4: F₃ = 0.1577 + (0.5)(0.4607) = 0.3880

p₃ = 1/(1 + e^−0.3880) = 0.5958

Loans 5–10: F₃ = -0.7820 + (0.5)(-0.3139) = -0.9390

p₃ = 1/(1 + e^−(-0.9390)) = 0.2811

6. Final Results

Loan	Y	F₀	F₁	F₂	F₃	Final p	Prediction
1	1	-0.4055	-0.1055	0.1577	0.3880	0.5958	Default
2	1	-0.4055	-0.1055	0.1577	0.3880	0.5958	Default
3	1	-0.4055	-0.1055	0.1577	0.3880	0.5958	Default
4	1	-0.4055	-0.1055	0.1577	0.3880	0.5958	Default
5	0	-0.4055	-0.6055	-0.7820	-0.9390	0.2811	No Default
6	0	-0.4055	-0.6055	-0.7820	-0.9390	0.2811	No Default
7	0	-0.4055	-0.6055	-0.7820	-0.9390	0.2811	No Default
8	0	-0.4055	-0.6055	-0.7820	-0.9390	0.2811	No Default
9	0	-0.4055	-0.6055	-0.7820	-0.9390	0.2811	No Default
10	0	-0.4055	-0.6055	-0.7820	-0.9390	0.2811	No Default

Classification rule: p ≥ 0.5 → Default; p < 0.5 → No Default.
For this deliberately simple dataset and the selected splits, the 3-tree model classifies all 10 training loans correctly.

7. Algorithm Summary

Step 1: p₀ = number of defaults / total loans = 0.40

Step 2: F₀ = ln(p₀/(1−p₀)) = -0.4055

Step 3: rᵢ = Yᵢ − pᵢ

Step 4: γ_leaf = Σ(Y − p) / N_leaf

Step 5: F_t = F_t−1 + ηγ, η = 0.5

Step 6: p_t = 1/(1 + e^−F_t)

Step 7: Repeat for Tree 2, Tree 3, …

Key distinction: F₀ is calculated using the logistic log-odds formula. The later leaf values are intentionally the simple average residual, not the Newton/Hessian leaf formula.

Gradient Boosting Classification • Loan Default Worked Example • Simple Gradient/Residual Leaf Values

USE CASE 1: Use Gradient Boosting with scikit-learn to predict whether a loan will default. Dependent variable: Default (0 = No Default, 1 = Default) Independent variables (3): Income (monthly income, e.g., 1000–10000) CreditScore (300–850) LoanAmount (1000–50000).

import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import GradientBoostingClassifier from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, roc_auc_score # ----------------------------------- # 1. Load data from Excel # ----------------------------------- data = pd.read_excel("loan_data.xlsx") df = pd.DataFrame(data) print("Dataset Preview:") print(data.head()) # ----------------------------------- # 2. Define features and target # ----------------------------------- X = df[["Income", "CreditScore", "LoanAmount"]] y = df["Default"] # ----------------------------------- # 3. Split into training and testing # ----------------------------------- X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42 ) gb_model = GradientBoostingClassifier( n_estimators=100, # number of trees learning_rate=0.1, # step size max_depth=3, # tree depth random_state=42 ) gb_model.fit(X_train, y_train) y_pred = gb_model.predict(X_test) y_prob = gb_model.predict_proba(X_test)[:, 1] print("Accuracy:", accuracy_score(y_test, y_pred)) print("\nConfusion Matrix:\n", confusion_matrix(y_test, y_pred)) print("\nClassification Report:\n", classification_report(y_test, y_pred)) print("\nROC-AUC Score:", roc_auc_score(y_test, y_prob)) Step 9: Predict default for a new customer new_customer = [[4500, 620, 16000]] # Income, CreditScore, LoanAmount default_prediction = gb_model.predict(new_customer) default_probability =gb_model.predict_proba(new_customer)[0][1] print("Default Prediction:", default_prediction[0]) print("Probability of Default:", default_probability)

import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import GradientBoostingClassifier from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, roc_auc_score # ----------------------------------- # 1. Load data from Excel # ----------------------------------- #sample data can be exported to #excel from the URL # https://www.pythonplaza.com/categorical_customer_churn_1_or_0.html data = pd.read_excel("customer_data.xlsx") print("Dataset Preview:") print(data.head()) # ----------------------------------- # 2. Define features and target # ----------------------------------- X = df[["Tenure", "MonthlyCharges", "ContractType", "SupportCalls"]] y = df["Churn"] # ----------------------------------- # 3. Split into training and testing # ----------------------------------- X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42 ) gb_model = GradientBoostingClassifier( n_estimators=100, # number of trees learning_rate=0.1, # step size max_depth=3, # tree depth random_state=42 ) gb_model.fit(X_train, y_train) y_pred = gb_model.predict(X_test) y_prob = gb_model.predict_proba(X_test)[:, 1] print("Accuracy:", accuracy_score(y_test, y_pred)) print("\nConfusion Matrix:\n", confusion_matrix(y_test, y_pred)) print("\nClassification Report:\n", classification_report(y_test, y_pred)) print("\nROC-AUC Score:", roc_auc_score(y_test, y_prob)) #Predict churn for a new customer new_customer = [[8, 92, 0, 5]] # Tenure, MonthlyCharges, ContractType, SupportCalls churn_prediction = gb_model.predict(new_customer) churn_probability = gb_model.predict_proba(new_customer)[0][1] print("Churn Prediction:", churn_prediction[0]) print("Probability of Churn:", churn_probability) #Interpreting the results (business view) 1 → High risk of churn ⚠️ 0 → Likely to stay ✅ Use probability (e.g., churn > 0.6) to trigger retention offers

import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import GradientBoostingClassifier from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, roc_auc_score # ----------------------------------- # 1. Load data from Excel # ----------------------------------- #sample data can be exported to #excel from the URL Get the Categorical learning Styles data in Excel data = pd.read_excel("Categorical_learning_Styles.xlsx") print("Dataset Preview:") print(data.head()) # ----------------------------------- # 2. Define the data # ----------------------------------- X = df[['prefers_diagrams', 'prefers_lectures', 'prefers_notes', 'prefers_hands_on']] y = df['learning_style'] # Encode categorical target labels le = LabelEncoder() y_encoded = le.fit_transform(y) # ----------------------------------- # 3. Split into training and testing # ----------------------------------- X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42 ) gb_model = GradientBoostingClassifier( n_estimators=100, # number of trees learning_rate=0.1, # step size max_depth=3, # tree depth random_state=42 ) gb_model.fit(X_train, y_train) y_pred = gb_model.predict(X_test) y_prob = gb_model.predict_proba(X_test)[:, 1] print("Accuracy:", accuracy_score(y_test, y_pred)) print("\nConfusion Matrix:\n", confusion_matrix(y_test, y_pred)) print("\nClassification Report:\n", classification_report(y_test, y_pred)) print("\nROC-AUC Score:", roc_auc_score(y_test, y_prob)) #Predict with sample data new_students = np.array([ [5, 1, 2, 1], # Likely Visual [1, 5, 3, 2], # Likely Auditory [2, 1, 5, 2], # Likely Reading/Writing [1, 2, 1, 5] # Likely Kinesthetic ]) # Predict encoded labels predictions_encoded = gb_model.predict(new_students) # Convert numeric predictions back to original labels predictions = le.inverse_transform(predictions_encoded) print("Predicted Learning Styles:") print(predictions)

import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import GradientBoostingClassifier from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, roc_auc_score # ----------------------------------- # 1. Load data from Excel # ----------------------------------- #sample data can be exported to #excel from the URL Get Disease Classification in Excel data = pd.read_excel("patient_dosage_response.xlsx") print("Dataset Preview:") print(data.head()) df = pd.DataFrame(data) # ---------------------------------- # 2. Separate Features and Target # ---------------------------------- X = df[['Age', 'BloodPressure', 'Cholesterol', 'FamilyHistory']] y = df['Disease'] # ----------------------------------- # 3. Split into training and testing # ----------------------------------- X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42 ) gb_model = GradientBoostingClassifier( n_estimators=100, # number of trees learning_rate=0.1, # step size max_depth=3, # tree depth random_state=42 ) gb_model.fit(X_train, y_train) y_pred = gb_model.predict(X_test) y_prob = gb_model.predict_proba(X_test)[:, 1] print("Accuracy:", accuracy_score(y_test, y_pred)) print("\nConfusion Matrix:\n", confusion_matrix(y_test, y_pred)) print("\nClassification Report:\n", classification_report(y_test, y_pred)) print("\nROC-AUC Score:", roc_auc_score(y_test, y_prob)) new_patients = np.array([ [45, 150, 230, 1], # High risk [28, 118, 175, 0] # Low risk ]) predictions = gb_model.predict(new_patients) print("Disease Predictions:") print(predictions)

Supervised Machine Learning Algorithms