Sliced Bank Account Churn

workflowr

• Summary
• Checks
• Past versions

Last updated: 2021-10-11

Checks: 7 0

Knit directory: myTidyTuesday/

This reproducible R Markdown analysis was created with workflowr (version 1.6.2). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

---

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20210907)

The command set.seed(20210907) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: 49b25bc

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version 49b25bc. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:

Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/
    Ignored:    catboost_info/
    Ignored:    data/2021-10-11/
    Ignored:    data/CNHI_Excel_Chart.xlsx
    Ignored:    data/CommunityTreemap.jpeg
    Ignored:    data/Community_Roles.jpeg
    Ignored:    data/YammerDigitalDataScienceMembership.xlsx
    Ignored:    data/accountchurn.rds
    Ignored:    data/acs_poverty.rds
    Ignored:    data/airbnbcatboost.rds
    Ignored:    data/australiaweather.rds
    Ignored:    data/fmhpi.rds
    Ignored:    data/grainstocks.rds
    Ignored:    data/hike_data.rds
    Ignored:    data/nber_rs.rmd
    Ignored:    data/netflixTitles.rmd
    Ignored:    data/netflixTitles2.rds
    Ignored:    data/us_states.rds
    Ignored:    data/us_states_hexgrid.geojson
    Ignored:    data/weatherstats_toronto_daily.csv

Untracked files:
    Untracked:  analysis/CHN_1_sp.rds
    Untracked:  analysis/sample data for r test.xlsx
    Untracked:  code/YammerReach.R
    Untracked:  code/work list batch targets.R

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

---

These are the previous versions of the repository in which changes were made to the R Markdown (analysis/2021_07_13_sliced.Rmd) and HTML (docs/2021_07_13_sliced.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.

File	Version	Author	Date	Message
Rmd	49b25bc	opus1993	2021-10-11	adopt common color palette and dotplots

---

SLICED is like the TV Show Chopped but for data science. Competitors get a never-before-seen dataset and two-hours to code a solution to a prediction challenge. Contestants get points for the best model plus bonus points for data visualization, votes from the audience, and more.

Season 1 Episode 7 features a challenge to predict whether a bank customer is churned (lost to another bank). The evaluation metric for submissions in this competition is residual mean log loss.

To make the best use of the resources that we have, we will explore the data set for features to select those with the most predictive power, build a random forest to confirm the recipe, and then use the rest of the time to build one or more ensemble models.

Let’s load up some packages:

suppressPackageStartupMessages({
library(tidyverse)
library(hrbrthemes)
library(lubridate)

library(tidymodels)
library(treesnip)
library(finetune)

library(themis)
library(baguette)
  
library(catboost)

})

source(here::here("code","_common.R"),
       verbose = FALSE,
       local = knitr::knit_global())


ggplot2::theme_set(theme_jim(base_size = 12))

#create a data directory
data_dir <- here::here("data",Sys.Date())
if (!file.exists(data_dir)) dir.create(data_dir)

# set a competition metric
mset <- metric_set(mn_log_loss)

# set the competition name from the web address
competition_name <- "sliced-s01e07-HmPsw2"

zipfile <- paste0(data_dir,"/", competition_name, ".zip")

path_export <- here::here("data",Sys.Date(),paste0(competition_name,".csv"))

Import and Exploratory Data Analysis

A quick reminder before downloading the dataset: Go to the web site and accept the competition terms!!!

Direct Import and Shell Commands

We have basic shell commands available to interact with Kaggle here:

# from the Kaggle api https://github.com/Kaggle/kaggle-api

# the leaderboard
shell(glue::glue("kaggle competitions leaderboard { competition_name } -s"))

# the files to download
shell(glue::glue("kaggle competitions files -c { competition_name }"))

# the command to download files
shell(glue::glue("kaggle competitions download -c { competition_name } -p { data_dir }"))

# unzip the files received
shell(glue::glue("unzip { zipfile } -d { data_dir }"))

Reading in the contents of the datafiles here:

train_df <-
  read_csv(file = glue::glue(
    {
      data_dir
    },
    "/train.csv"
  )) %>%
  mutate(across(
    c(attrition_flag, gender, education_level, income_category),
    as.factor
  ))

test_df <-
  read_csv(file = glue::glue(
    {
      data_dir
    },
    "/test.csv"
  )) %>%
  mutate(across(c(gender, education_level, income_category), as.factor))

Skim

Some questions to answer here: What features have missing data, and imputations may be required? What does the outcome variable look like, in terms of imbalance?

skimr::skim(train_df)

Outcome variable attrition_flag is set to 1 (churned) for 1132 rows, and 0 (not churned) for 5956, so is imbalanced. None of the features is missing obvious data. We will take a closer look at the categorical variable levels in a moment.

Categorical Feature Plots

summarize_attrition <- function(tbl) {
  ret <- tbl %>%
    summarize(
      n_attrition = sum(attrition_flag == 1),
      n = n(),
      .groups = "drop"
    ) %>%
    arrange(desc(n)) %>%
    mutate(
      pct_attrition = n_attrition / n,
      low = qbeta(.025, n_attrition + 5, n - n_attrition + .5),
      high = qbeta(.975, n_attrition + 5, n - n_attrition + .5)
    ) %>%
    mutate(pct = n / sum(n))
  ret
}

train_df %>%
  summarize_attrition()

train_df %>%
  group_by(education_level) %>%
  summarize_attrition() %>%
  mutate(education_level = fct_reorder(education_level, pct_attrition)) %>%
  ggplot(aes(pct_attrition, education_level)) +
  geom_point(aes(size = pct)) +
  geom_errorbarh(aes(xmin = low, xmax = high), height = .3) +
  scale_size_continuous(
    labels = percent,
    guide = "none",
    range = c(.5, 4)
  ) +
  scale_x_continuous(labels = percent) +
  labs(
    x = "Proportion of attrition",
    y = "",
    title = "What education levels get the most attrition?",
    subtitle = "Including 95% intervals. Size of points is proportional to frequency in the dataset"
  )

train_df %>%
  group_by(income_category = fct_lump(income_category, 30)) %>%
  summarize_attrition() %>%
  mutate(income_category = fct_reorder(income_category, pct_attrition)) %>%
  ggplot(aes(pct_attrition, income_category)) +
  geom_point(aes(size = pct)) +
  geom_errorbarh(aes(xmin = low, xmax = high), height = .3) +
  scale_size_continuous(
    labels = percent,
    guide = "none",
    range = c(.5, 4)
  ) +
  scale_x_continuous(labels = percent) +
  labs(
    x = "Proportion of attrition",
    y = "",
    title = "What income category levels get the most attrition?",
    subtitle = "Including 95% confidence intervals. Size of points is proportional to frequency in the dataset"
  )

train_df %>%
  ggplot(aes(credit_limit, gender, color = attrition_flag)) +
  ggdist::stat_dots(
    aes(fill = attrition_flag),
    side = "top",
    alpha = 0.2,
    justification = -0.1,
    binwidth = 100,
    dotsize = 1,
    stackratio = 0.6,
    show.legend = FALSE
  ) +
  geom_boxplot(
    width = 0.1,
    outlier.shape = NA,
    show.legend = FALSE
  ) +
  scale_x_continuous(labels = scales::dollar_format(accuracy = 1)) +
  theme(
    plot.subtitle = ggtext::element_textbox_simple(),
    plot.background = element_rect(color = "white"),
    panel.grid.major.y = element_blank()
  ) +
  labs(
    title = "Differences in Credit Limits, by Gender",
    subtitle = "Customers <span style = 'color:#F1CA3AFF'>churning</span> and <span style = 'color:#7A0403FF'> not churning</span>. Note that often men have higher limits.",
    y = "Gender",
    x = "Credit Limit"
  )

Numeric Feature Plots

The outcome variable itself is skewed across all observations in the training data, as prices often are.

train_numeric <- train_df %>%
  keep(is.numeric) %>%
  colnames()

chart <- c(train_numeric, "attrition_flag")

train_df %>%
  select_at(vars(all_of(chart))) %>%
  select(-id) %>%
  pivot_longer(
    cols = -attrition_flag,
    names_to = "key",
    values_to = "value"
  ) %>%
  filter(!is.na(value)) %>%
  ggplot(mapping = aes(value,
    #                     after_stat(density),
    fill = attrition_flag
  )) +
  geom_histogram(
    position = "identity",
    alpha = 0.5,
    bins = 30,
    show.legend = FALSE
  ) +
  facet_wrap(~key, scales = "free", ncol = 3) +
  scale_x_continuous(n.breaks = 3) +
  theme(
    plot.subtitle = ggtext::element_textbox_simple(),
    plot.background = element_rect(color = "white")
  ) +
  labs(
    title = "Numeric Feature Histogram Distributions",
    subtitle = "Customers <span style = 'color:#F1CA3AFF'>churning</span> and <span style = 'color:#7A0403FF'> not churning</span>",
    x = "Numeric Feature",
    y = NULL
  )

Numeric Feature Correlations

cr1 <- train_df %>%
  keep(is.numeric) %>%
  cor(use = "pair")

corrplot::corrplot(cr1, type = "upper")

The transaction amounts and transaction counts tend to be correlated. Also, the revolving balance and average utilization ratio.

---

Conclusions:

• There aren’t any obvious features to engineer, so as a first pass we will go straight to training hyperparameters with the features we have.
• As a second pass, we can add interactions to try to improve the model.

Preprocessing

The recipe framework

I am going to train on almost all of the train data. The 1% left is a quick confirmation of validity. The provided file labeled test has no labels.

set.seed(2021)
split <- train_df %>%
  initial_split(
    strata = attrition_flag,
    prop = .99
  )
train <- training(split)
valid <- testing(split)

There are only 72 held out from training as a last check before submission.

basic_rec <-
  recipe(attrition_flag ~ ., train) %>%
  # ---- set aside the row id's
  update_role(id, new_role = "id") %>%
  step_novel(gender, education_level, income_category) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_nzv(all_predictors())

the pre-processed data

basic_rec %>%
  #  finalize_recipe(list(num_comp = 2)) %>%
  prep() %>%
  juice()

Cross Validation

We will use 5-fold cross validation and stratify between the churn and no-churn classes.

train_folds <- vfold_cv(
  data = train,
  strata = attrition_flag,
  v = 5
)

Machine Learning

Let’s run models in two steps. The first is a simple, fast shallow random forest, to confirm that the model will run and observe feature importance scores. The second will use catboost. Both use the basic recipe preprocessor.

Model Specifications

We will build a specification for simple shallow random forest and a specification for catboost.

catboost_spec <- boost_tree(
  trees = 1000,
  min_n = tune(),
  learn_rate = tune(),
  tree_depth = tune()
) %>%
  set_engine("catboost") %>%
  set_mode("classification")

bag_spec <-
  bag_tree(min_n = 10) %>%
  set_engine("rpart", times = 50) %>%
  set_mode("classification")

Parallel backend

To speed up computation we will use a parallel backend.

all_cores <- parallelly::availableCores(omit = 1)
all_cores

system 
    11 

future::plan("multisession", workers = all_cores) # on Windows

A quick Random Forest

Lets make a cursory check of the recipe and variable importance, which comes out of rpart for free. This workflow also handles factors without dummies.

bag_wf <-
  workflow() %>%
  add_recipe(basic_rec) %>%
  add_model(bag_spec)

set.seed(123)
bag_fit <- parsnip::fit(bag_wf, data = train)

extract_fit_parsnip(bag_fit)$fit$imp %>%
  mutate(term = fct_reorder(term, value)) %>%
  ggplot(aes(value, term)) +
  geom_point() +
  geom_errorbarh(aes(
    xmin = value - `std.error` / 2,
    xmax = value + `std.error` / 2
  ),
  height = .3
  ) +
  labs(
    title = "Feature Importance",
    x = NULL, y = NULL
  )

We see that total_trans_ct and total_trans_amt will be very important for this model.

Tuning Catboost

Now that we have some confidence that the features have predictive power, lets tune up a set of catboost models.

catboost_params <-
  dials::parameters(
    min_n(), # min data in leaf
    tree_depth(range = c(4, 15)),
    learn_rate(
      range = c(-3, -0.7),
      trans = log10_trans()
    )
  )

cbst_grid <- dials::grid_max_entropy(catboost_params,
  size = 40
)
cbst_grid

cv_res_catboost <-
  workflow() %>%
  add_recipe(basic_rec) %>%
  add_model(catboost_spec) %>%
  tune_grid(
    resamples = train_folds,
    grid = cbst_grid,
    control = control_race(
      verbose = FALSE,
      save_pred = TRUE,
      save_workflow = TRUE,
      extract = extract_model,
      parallel_over = "resamples"
    ),
    metrics = mset
  )

autoplot(cv_res_catboost)

show_best(cv_res_catboost) %>%
  select(-.estimator)

cat_wf_best <-
  workflow() %>%
  add_recipe(basic_rec) %>%
  add_model(catboost_spec) %>%
  finalize_workflow(select_best(cv_res_catboost))

cat_fit_best <- cat_wf_best %>%
  parsnip::fit(data = train)

Catboost model performance

predict(cat_fit_best, new_data = valid, type = "prob") %>%
  cbind(valid) %>%
  mn_log_loss(attrition_flag, `.pred_0`)

This catboost mean logloss figure is not bad, so I’m going to write it out in a suitable csv…

bind_cols(predict(cat_fit_best, test_df), test_df) %>%
  select(id, attrition_flag = `.pred_class`) %>%
  write_csv(file = path_export)

and make the submission to the Kaggle board

shell(glue::glue('kaggle competitions submit -c { competition_name } -f { path_export } -m "Catboosted with numeric interactions"'))

Machine Learning Round 2

Recipe with Interactions

Now modeling with the numeric interaction features, tightening up the learn rate, and reducing the size of the grid to speed up results. We are nearly out of time.

advanced_rec <-
  recipe(attrition_flag ~ ., train) %>%
  # ---- set aside the row id's
  update_role(id, new_role = "id") %>%
  step_novel(gender, education_level, income_category) %>%
  step_interact(terms = ~ total_relationship_count:avg_utilization_ratio) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_nzv(all_predictors())

Advanced Catboost Tuning

catboost_params <-
  dials::parameters(
    min_n(), # min data in leaf
    tree_depth(range = c(4, 15)),
    learn_rate(
      range = c(-1, -0.7), # updated
      trans = log10_trans()
    )
  )

cbst_grid <- dials::grid_max_entropy(catboost_params,
  size = 20
)
cbst_grid

cv_res_catboost <-
  workflow() %>%
  add_recipe(advanced_rec) %>%
  add_model(catboost_spec) %>%
  tune_grid(
    resamples = train_folds,
    grid = cbst_grid,
    control = control_race(
      verbose = FALSE,
      save_pred = TRUE,
      save_workflow = TRUE,
      extract = extract_model,
      parallel_over = "resamples"
    ),
    metrics = mset
  )

autoplot(cv_res_catboost)

show_best(cv_res_catboost) %>%
  select(-.estimator)

cat_wf_best <-
  workflow() %>%
  add_recipe(basic_rec) %>%
  add_model(catboost_spec) %>%
  finalize_workflow(select_best(cv_res_catboost))

cat_fit_best <- cat_wf_best %>%
  parsnip::fit(data = train)

Advanced catboost model performance

predict(cat_fit_best, new_data = valid, type = "prob") %>%
  cbind(valid) %>%
  mn_log_loss(attrition_flag, `.pred_0`)

This catboost figure is not an improvement, so I’m not going to submit.

Session information

sessionInfo()

R version 4.1.1 (2021-08-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] vctrs_0.3.8         rlang_0.4.11        catboost_0.26      
 [4] baguette_0.1.1      themis_0.1.4        finetune_0.1.0     
 [7] treesnip_0.1.0.9000 yardstick_0.0.8     workflowsets_0.1.0 
[10] workflows_0.2.3     tune_0.1.6          rsample_0.1.0      
[13] recipes_0.1.17      parsnip_0.1.7.900   modeldata_0.1.1    
[16] infer_1.0.0         dials_0.0.10        scales_1.1.1       
[19] broom_0.7.9         tidymodels_0.1.4    lubridate_1.7.10   
[22] hrbrthemes_0.8.0    forcats_0.5.1       stringr_1.4.0      
[25] dplyr_1.0.7         purrr_0.3.4         readr_2.0.2        
[28] tidyr_1.1.4         tibble_3.1.4        ggplot2_3.3.5      
[31] tidyverse_1.3.1     workflowr_1.6.2    

loaded via a namespace (and not attached):
  [1] utf8_1.2.2           R.utils_2.11.0       tidyselect_1.1.1    
  [4] grid_4.1.1           pROC_1.18.0          munsell_0.5.0       
  [7] codetools_0.2-18     ragg_1.1.3           future_1.22.1       
 [10] withr_2.4.2          colorspace_2.0-2     highr_0.9           
 [13] knitr_1.36           rstudioapi_0.13      Rttf2pt1_1.3.8      
 [16] listenv_0.8.0        labeling_0.4.2       git2r_0.28.0        
 [19] TeachingDemos_2.12   farver_2.1.0         bit64_4.0.5         
 [22] DiceDesign_1.9       rprojroot_2.0.2      mlr_2.19.0          
 [25] parallelly_1.28.1    generics_0.1.0       ipred_0.9-12        
 [28] xfun_0.26            markdown_1.1         R6_2.5.1            
 [31] doParallel_1.0.16    lhs_1.1.3            cachem_1.0.6        
 [34] assertthat_0.2.1     promises_1.2.0.1     vroom_1.5.5         
 [37] nnet_7.3-16          gtable_0.3.0         Cubist_0.3.0        
 [40] globals_0.14.0       timeDate_3043.102    BBmisc_1.11         
 [43] systemfonts_1.0.2    splines_4.1.1        butcher_0.1.5       
 [46] extrafontdb_1.0      earth_5.3.1          checkmate_2.0.0     
 [49] yaml_2.2.1           reshape2_1.4.4       modelr_0.1.8        
 [52] backports_1.2.1      httpuv_1.6.3         gridtext_0.1.4      
 [55] extrafont_0.17       usethis_2.0.1        inum_1.0-4          
 [58] tools_4.1.1          lava_1.6.10          ellipsis_0.3.2      
 [61] jquerylib_0.1.4      Rcpp_1.0.7           plyr_1.8.6          
 [64] parallelMap_1.5.1    rpart_4.1-15         ParamHelpers_1.14   
 [67] viridis_0.6.1        haven_2.4.3          fs_1.5.0            
 [70] here_1.0.1           furrr_0.2.3          unbalanced_2.0      
 [73] magrittr_2.0.1       data.table_1.14.2    ggdist_3.0.0        
 [76] reprex_2.0.1         RANN_2.6.1           GPfit_1.0-8         
 [79] mvtnorm_1.1-2        whisker_0.4          ROSE_0.0-4          
 [82] R.cache_0.15.0       hms_1.1.1            evaluate_0.14       
 [85] readxl_1.3.1         gridExtra_2.3        compiler_4.1.1      
 [88] crayon_1.4.1         R.oo_1.24.0          htmltools_0.5.2     
 [91] later_1.3.0          tzdb_0.1.2           Formula_1.2-4       
 [94] ggtext_0.1.1         libcoin_1.0-9        DBI_1.1.1           
 [97] corrplot_0.90        dbplyr_2.1.1         MASS_7.3-54         
[100] Matrix_1.3-4         cli_3.0.1            C50_0.1.5           
[103] R.methodsS3_1.8.1    parallel_4.1.1       gower_0.2.2         
[106] pkgconfig_2.0.3      xml2_1.3.2           foreach_1.5.1       
[109] bslib_0.3.0          hardhat_0.1.6        plotmo_3.6.1        
[112] prodlim_2019.11.13   rvest_1.0.1          distributional_0.2.2
[115] digest_0.6.28        rmarkdown_2.11       cellranger_1.1.0    
[118] fastmatch_1.1-3      gdtools_0.2.3        lifecycle_1.0.1     
[121] jsonlite_1.7.2       viridisLite_0.4.0    fansi_0.5.0         
[124] pillar_1.6.3         lattice_0.20-44      fastmap_1.1.0       
[127] httr_1.4.2           plotrix_3.8-2        survival_3.2-11     
[130] glue_1.4.2           conflicted_1.0.4     FNN_1.1.3           
[133] iterators_1.0.13     bit_4.0.4            class_7.3-19        
[136] stringi_1.7.5        sass_0.4.0           rematch2_2.1.2      
[139] textshaping_0.3.5    partykit_1.2-15      styler_1.6.2        
[142] future.apply_1.8.1