AI summary by TwinMind
XGBoost in Python from Start to Finish
StatQuest with Josh Starmer
56m 43s
Introduction and Setup
- Webinar on XGBoost in Python from start to finish
- Using Jupyter notebook for hands-on tutorial
- Predicting customer churn using IBM telco dataset
- XGBoost builds collection of boosted trees for classification
- Prerequisites: Python basics, XGBoost theory, cross validation, confusion matrices
- Required modules: pandas, numpy, scikit-learn, XGBoost
- Optional: graphviz for tree visualization
Data Loading and Initial Exploration
- Loading telco churn dataset from IBM base samples
- Dataset contains 33 columns, 5 rows displayed initially
- Using pandas read_csv() function
- Data organized in rows and columns (spreadsheet format)
- Standard variable name 'df' for dataframe
Data Cleaning and Preprocessing
- Removing irrelevant columns: churn_reason, CLTV, churn_score (exit interview data)
- These provide perfect prediction but unavailable before churn
- Identifying single-value columns: count (only 1s), country (only "United States"), state (only "California")
- Removing non-predictive columns: customer_id, lat_long (redundant with separate lat/long columns)
- Final dataset: 24 columns after cleaning
- Replacing whitespace with underscores in city names and column names
- Required for tree visualization, not XGBoost functionality
Missing Data Identification and Handling
- XGBoost has default behavior for missing data - sets to zero
- Checking data types using dtypes() function
- Most columns are "object" type (text responses like yes/no)
- Identified issue: total_charges column shows as object despite numeric appearance
- Found blank spaces in total_charges column (11 rows with missing values)
- All missing total_charges correspond to tenure_months = 0 (new customers)
- Set missing total_charges values to zero
- Converted total_charges from object to float64 type
- Replaced remaining whitespace with underscores dataframe-wide
Data Formatting for XGBoost
- Split data into features (X) and target (y)
- X: all columns except churn_value
- y: churn_value column (1 for left, 0 for stayed)
- Verified y contains only 0s and 1s
One-Hot Encoding Process
- XGBoost supports continuous data natively but requires encoding for categorical data
- Categorical columns need conversion to format XGBoost can use
- Example: payment_method has 4 categories (mailed_check, electronic_check, bank_transfer, credit_card)
- Converting categories to numbers (1,2,3,4) assumes similarity/ordering that doesn't exist
- One-hot encoding creates separate binary column for each category
- Using pandas get_dummies() function
- Result: 1,178 columns after encoding (many city names)
- Each category gets 1 if true, 0 if false
Mini StatQuest: One-Hot Encoding with Missing Data
- Example: favorite_color column with blue, green, and missing values
- Missing values coded as zeros
- One-hot encoding creates blue and green columns
- Missing data gets zeros in both columns
- XGBoost determines optimal clustering of missing data with existing categories
- Uses sparse matrices - only tracks 1s, doesn't allocate memory for 0s
- Memory efficient approach for handling missing data
Train-Test Split and Data Imbalance
- Dataset is imbalanced: only 27% of customers left (churn_value = 1)
- Using stratified split to maintain same percentage in train/test sets
- train_test_split with stratify=y parameter
- Verified: 27% churn rate in both training and testing sets
Initial XGBoost Model Building
- Using XGBClassifier with binary:logistic objective
- Setting missing=None (default behavior for zeros)
- Random state = 42 for reproducibility
- Early stopping: stops if no improvement after 10 consecutive trees
- Evaluation metric: AUC (Area Under Curve)
- Model stopped after 45 trees (best iteration)
- Training on training set, evaluating stopping criteria on test set
Initial Model Performance
- Confusion matrix results:
- People who stayed: 1,178/1,294 correctly classified (91%)
- People who left: 239/467 correctly classified (51%)
- Poor performance on minority class (people who left)
- Imbalanced data causing bias toward majority class
Parameter Optimization
- XGBoost has many hyperparameters: max_depth, learning_rate (eta), gamma, reg_lambda
- Using scale_pos_weight parameter to handle class imbalance
- Grid search with cross-validation to find optimal parameters
- Two-round optimization process (10 minutes runtime)
- Optimized parameters:
- max_depth: 4
- learning_rate: 0.1
- gamma: 0.25
- reg_lambda: 10
- scale_pos_weight: 3
- Using random subsampling (50% of data and features per tree) to speed up training
Optimized Model Performance
- Final model built with optimized parameters
- Early stopping after 55 trees (65 total built)
- Improved confusion matrix:
- People who left: 390/467 correctly classified (84% vs previous 51%)
- Trade-off: slightly worse performance on people who stayed
- Business perspective: Better to catch potential churners and send retention offers
- Acceptable to send unnecessary retention offers to loyal customers
Tree Visualization and Interpretation
- Built single tree for visualization (num_boost_round=1)
- Root node splits on contract_month_to_month < 1
- Tree shows feature names and thresholds
- Leaves provide probability increments (not final classifications)
- XGBoost combines all trees for final prediction
- Tree helps understand feature importance and decision boundaries
Conclusion
- Successfully completed end-to-end XGBoost pipeline:
- Data loading and cleaning
- Missing data handling
- One-hot encoding for categorical variables
- Model building and optimization
- Performance evaluation and interpretation
- Improved churn prediction from 51% to 84% accuracy on minority class
- Demonstrated XGBoost's handling of missing data and class imbalance
