AI summary by TwinMind

XGBoost in Python from Start to Finish

StatQuest with Josh Starmer

56m 43s

Introduction and Setup

  • Webinar on XGBoost in Python from start to finish
  • Using Jupyter notebook for hands-on tutorial
  • Predicting customer churn using IBM telco dataset
  • XGBoost builds collection of boosted trees for classification
  • Prerequisites: Python basics, XGBoost theory, cross validation, confusion matrices
  • Required modules: pandas, numpy, scikit-learn, XGBoost
  • Optional: graphviz for tree visualization

Data Loading and Initial Exploration

  • Loading telco churn dataset from IBM base samples
  • Dataset contains 33 columns, 5 rows displayed initially
  • Using pandas read_csv() function
  • Data organized in rows and columns (spreadsheet format)
  • Standard variable name 'df' for dataframe

Data Cleaning and Preprocessing

  • Removing irrelevant columns: churn_reason, CLTV, churn_score (exit interview data)
    • These provide perfect prediction but unavailable before churn
  • Identifying single-value columns: count (only 1s), country (only "United States"), state (only "California")
  • Removing non-predictive columns: customer_id, lat_long (redundant with separate lat/long columns)
  • Final dataset: 24 columns after cleaning
  • Replacing whitespace with underscores in city names and column names
    • Required for tree visualization, not XGBoost functionality

Missing Data Identification and Handling

  • XGBoost has default behavior for missing data - sets to zero
  • Checking data types using dtypes() function
  • Most columns are "object" type (text responses like yes/no)
  • Identified issue: total_charges column shows as object despite numeric appearance
  • Found blank spaces in total_charges column (11 rows with missing values)
  • All missing total_charges correspond to tenure_months = 0 (new customers)
  • Set missing total_charges values to zero
  • Converted total_charges from object to float64 type
  • Replaced remaining whitespace with underscores dataframe-wide

Data Formatting for XGBoost

  • Split data into features (X) and target (y)
  • X: all columns except churn_value
  • y: churn_value column (1 for left, 0 for stayed)
  • Verified y contains only 0s and 1s

One-Hot Encoding Process

  • XGBoost supports continuous data natively but requires encoding for categorical data
  • Categorical columns need conversion to format XGBoost can use
  • Example: payment_method has 4 categories (mailed_check, electronic_check, bank_transfer, credit_card)
  • Converting categories to numbers (1,2,3,4) assumes similarity/ordering that doesn't exist
  • One-hot encoding creates separate binary column for each category
  • Using pandas get_dummies() function
  • Result: 1,178 columns after encoding (many city names)
  • Each category gets 1 if true, 0 if false

Mini StatQuest: One-Hot Encoding with Missing Data

  • Example: favorite_color column with blue, green, and missing values
  • Missing values coded as zeros
  • One-hot encoding creates blue and green columns
  • Missing data gets zeros in both columns
  • XGBoost determines optimal clustering of missing data with existing categories
  • Uses sparse matrices - only tracks 1s, doesn't allocate memory for 0s
  • Memory efficient approach for handling missing data

Train-Test Split and Data Imbalance

  • Dataset is imbalanced: only 27% of customers left (churn_value = 1)
  • Using stratified split to maintain same percentage in train/test sets
  • train_test_split with stratify=y parameter
  • Verified: 27% churn rate in both training and testing sets

Initial XGBoost Model Building

  • Using XGBClassifier with binary:logistic objective
  • Setting missing=None (default behavior for zeros)
  • Random state = 42 for reproducibility
  • Early stopping: stops if no improvement after 10 consecutive trees
  • Evaluation metric: AUC (Area Under Curve)
  • Model stopped after 45 trees (best iteration)
  • Training on training set, evaluating stopping criteria on test set

Initial Model Performance

  • Confusion matrix results:
    • People who stayed: 1,178/1,294 correctly classified (91%)
    • People who left: 239/467 correctly classified (51%)
  • Poor performance on minority class (people who left)
  • Imbalanced data causing bias toward majority class

Parameter Optimization

  • XGBoost has many hyperparameters: max_depth, learning_rate (eta), gamma, reg_lambda
  • Using scale_pos_weight parameter to handle class imbalance
  • Grid search with cross-validation to find optimal parameters
  • Two-round optimization process (10 minutes runtime)
  • Optimized parameters:
    • max_depth: 4
    • learning_rate: 0.1
    • gamma: 0.25
    • reg_lambda: 10
    • scale_pos_weight: 3
  • Using random subsampling (50% of data and features per tree) to speed up training

Optimized Model Performance

  • Final model built with optimized parameters
  • Early stopping after 55 trees (65 total built)
  • Improved confusion matrix:
    • People who left: 390/467 correctly classified (84% vs previous 51%)
    • Trade-off: slightly worse performance on people who stayed
  • Business perspective: Better to catch potential churners and send retention offers
  • Acceptable to send unnecessary retention offers to loyal customers

Tree Visualization and Interpretation

  • Built single tree for visualization (num_boost_round=1)
  • Root node splits on contract_month_to_month < 1
  • Tree shows feature names and thresholds
  • Leaves provide probability increments (not final classifications)
  • XGBoost combines all trees for final prediction
  • Tree helps understand feature importance and decision boundaries

Conclusion

  • Successfully completed end-to-end XGBoost pipeline:
    • Data loading and cleaning
    • Missing data handling
    • One-hot encoding for categorical variables
    • Model building and optimization
    • Performance evaluation and interpretation
  • Improved churn prediction from 51% to 84% accuracy on minority class
  • Demonstrated XGBoost's handling of missing data and class imbalance

TwinMind

Get summaries like this for your own meetings

TwinMind takes notes in the background during meetings, lectures, and calls, then writes the summary for you.

Free plan to get started

No bot joins your calls

Phone, desktop & Chrome apps

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome

Your next meeting, summarized like this

TwinMind listens in the background and turns meetings, lectures, and calls into searchable notes you can chat with. Start free.

Available on phone, desktop & Chrome