Modeling - Decision Trees

Overview

Decision Trees are a heuristic based classification model which are useful for capturing non-linear trends and patterns in data. The heuristic aspect means that it follows a set of rules to provide an answer, whereas an algorithm follows steps to provide an answer which is always optimal. The tree aspect comes from the flowchart-like structure which features nodes and branches depending on decisions calculated from the data, constructing a logical pathway for classification.

The example above, from this [documentation], shows how decision trees can partition non-linear data. This is two-dimensional dataset example, but the same idea holds true for higher dimensions.

The example above, from this [website], shows the flowchart like structure, illustrating how a decision can be made by splitting logically on a criteria.

Notice that this example uses both qualitative and quantitative data. Decision Trees are effective on even mixed data. In fact, given at least a single column of quantitivate data, there are an infinite number of trees that can be made depending on how the quantitiatve variables are split. In addition to an infinite number of ways to split the quantitative variables (especially continuous data), tree depth can add to the complexity of a model.

Trees can be shallow or deep, meaning the number of branches and subsequent nodes that are allowed. A tree can be split until each node is pure or even only contains a single value. Purity in a node refers to the amount of labels within it. For example, given a decision tree whose task is to model a binary label dataset, if a node contains only a single label type, it is considered pure. Investigating the purity (and impurity) of a node is how "goodness" of split is measured. How are criteria for a split formed and how does this relate to purity? The common heuristics that are used in this process are:

• Gini
• Entropy
• Information Gain

Gini and Entropy calculate the impurity of a node, and Information Gain measures the overall impurity after a split is made and either Gini or Impurity is calculated. The attribute with the highest Information Gain is chosen for the split.

These formulas may seem complicated, but this can be illustrated better with an example using Entropy.

Decision Tree Example

In this small dataset, the data consists of Age, Blood Pressure, and Cholesterol, while the label is if a patient has a certaint disease (what the decision tree will try to predict).

Behind the scenes, the process that takes place is:

1. Entropy of the entire dataset is calculated (i.e. the 2 classes for every single row)
2. Entropy for each potential split is calculated
3. Information Gain is calculated via subtracting the Split Entropy from the System Entropy
4. The split providing the greatest Information Gain is taken.

In this example, the nodes are pure after the first split. If they weren't pure the process would continue with each node acting like the total system entropy and each potential split within that node being calculated to find the maximum information gain. Note that is a greedy algorithm, and takes the best possible result at each step and continues on.

Data Preparation

The decision tree portion of this project will utitlize the Weather and Ski Resorts data. It will use monthly averaged weather for each resort, and encoded regions, with the label being Pass type. As a reminder, the possible Pass types are Ikon, Epic, and Other. However, the Other category is overwhelming, and will be dropped. Decision Trees will be trained on Ikon and Epic and then will be used to find how well the Other categories would fit in either of the passes. In other words, possible expansion resorts for the popular passes.

Initial Data

Following from the previous sections, this section will use the same datasets. As a reminder, here is a preview of the data that will be used:

Ski Resorts Data

Weather Data

Prepared Data

Snippet of Complete Data

Snippet of Training Data

Snippet of Testing Data

Additionally, training and testing sets were created. The two sets are disjoint, and must be disjoint. Using non-disjoint data between testing and training won't give an accurate representation of the performance of the model. First, this could result in an overfit of the model, which could end up describing noise, rather than the underlying distribution. Second, the testing set being non-disjoint helps to represent real-world data (i.e. unseen data).

Also, the balance of the labels were examined, which is featured below:

Code

The code for preparing the data and running the decision tree modeling can be found [here].

Different hyperparameters were ran on the decision tree used to predict the label of Pass type. The goal was to see how the parameters changed (if possible) the root node. The different scenarios test the different parameters of scikit-learn's DecisionTreeClassifier().

Results

Scenario 1: Default Parameters

The default hyperparameters were:

• criterion: gini
• splitter: best
• class_weight: None

The default case resulted in an accuracy of 87.76% with the head node having the criteria Elevation High <= 1148.0.

Scenario 2: Change Entropy

The updated hyperparameters were:

• criterion: entropy
• splitter: best
• class_weight: None

This scenario used entropy instead of gini shows in an increase of accuracy, up to 94.69% with the head node having the criteria Elevation High <= 397.5.

Scenario 3: Change Log Loss

The updated hyperparameters were:

• criterion: log_loss
• splitter: best
• class_weight: None

This scenario used log_loss instead of gini shows in an increase of accuracy, up to 95.51% with the head node having the criteria Elevation High <= 397.5. Interestingly enough, this is the same root node as the scenario which used entropy instead of gini.

Scenario 4: Change Log Loss and Balance

The updated hyperparameters were:

• criterion: log_loss
• splitter: best
• class_weight: balanced

This scenario used log_loss instead of gini, and used balance instead of none for the weighting. This resulting in an accuracy of 97.96% with the head node having the criteria Elevation High <= 1148.0. Although this scenario increased the accuracy again, the root node has the same as the default scenario.

Scenario 5: Change Log Loss, Balance, and Randomize

The updated hyperparameters were:

• criterion: log_loss
• splitter: random
• class_weight: balanced

This scenario used log_loss instead of gini, balance instead of none for the weighting, and random instead of best for the splitter. This resulting in an accuracy of 91.84% with the head node having the criteria Region_Midwest <= 0.624. This scenario resulted in the third different root node.

Applications

With the high accuracies in these models, this could be a potential decent application. Recall that there are actually three pass types in the data, but the Other has been dropped for balancing measures. However, this provides an opportunity to illustrate possible expansions for the Ikon and Epic companies. To accomplish this, a model was trained with sklearn's GridSearchCV to test different parameters through cross fold validation. The different combinations of parameters tested above were input into the model.

The best parameters resulted in:

• criterion: entropy
• splitter: beest
• class_weight: balanced
• accuracy: 97.96%
• root node: Elevation High <= 1148.0

The rows containing the label of Pass type Other were originally dropped. Predictions were made on these dropped rows. The result of this was a label prediction for each month for each resort, thus the percentage of each label for each resort was examined. If a resort had a high percentage of being categorized as a certain pass over all its months, this would likely indicate that it could be a logical expansion.

The following table shows the percentage of time the resort was classified as either label.

The maximum percentage for each resort was taken as the label prediction, and a distribution of this maximum percentage was created.

Conclusions

This analysis provided valuable insights into potential expansions zones for the the Ikon and Epic ski resort pass companies. Provided common average monthly meteorological parameters and the region in which a potential expansion resort is located, a logical decision flowchart can be generated. The applications of this analysis can provide highly accurate indication if either company, Ikon or Epic, would find the resort a suitable potential expansion resort. This is currently based on weather and rough location, that being regional scale. However, further analyses investigating the culture, the amenities, and transportation would be recommended.