Modeling - Naive Bayes

Overview

Naive Bayes is a generative supervised machine learning classification method. The generative aspect indicates that the model learns from the probability of the data given previous knowledge of the label. This is in comparison to a discriminative model, where the goal is to find a function which distinguishes between groups (i.e. Logistic Regression). The supervised aspect means that the model is given labels to learn from. The Naïve aspect comes from the assumption that the categories have conditional independence in order to apply the Bayes’ Theorem. This is a Naive assumption because it’s unlikely that the variables within the data have true independence. For example, consider a model built from customer reviews which is trying to classify if the review was positive or negative. The review might have language such as “happy” and “glad”, which are clearly not independent terms. However, the assumption of independence is made to allow for the calculations to work. This example is one of sentiment analysis, however there are many potential applications of this method as it is implicitly acceptable for n-class classification. For example, predicting weather a label is “true” or “false” is a 2-class problem, but Naive Bayes can extend this to multiple labels. Back to the sentiment analysis example, a review could be “positive”, “negative”, or “neutral”. Applications also include document classification, which could be used to classify an article into categories such as “politics”, “sports”, “entertainment”, among many other overall article types.

In general, Naive Bayes uses the conditional independence assumption to apply the Bayes’ Theorem. Essentially, the goal is to find the probability of a label given a datapoint. The Bayes’ Theorem is appropriate for this task and uses several components of the probabilities within the data to calculate this. Namely, the probability of the data itself occurring, the probability of the label occurring, and the conditional probability in the opposite direction (i.e. the probability of the datapoint given a label). Especially in larger datasets, some of the conditional probabilities can be zero. This presents an issue due to the multiplicative calculations required, which would zero out the entire probability. Smoothing techniques are used to account for this, with the Laplacian Correction being the most common. This technique adds 1 to each case’s count. The general smoothing technique adds a specified variable (or alpha) to the count.

There are several primary forms of Naive Bayes:

• Multinomial Naive Bayes
• Gaussian Naive Bayes
• Bernoulli Naive Bayes
• Categorical Naive Bayes

Multinomial Naive Bayes

Multinomial Naive Bayes "is suitable for classification within discrete features (e.g., word counts for text classification)."[sklearn documentation]. Other discrete feature applications could be spam detection, sentiment analysis, and document categorization. Overall, this is a very efficient naive bayes method for text-based tasks.

Gaussian Naive Bayes

Gaussian Naive Bayes can be used when features are continuous, and are assumed to follow a normal (or Gaussian) distribution. Some of the specific applications could be medical diagnoses, real-time classification, and anomaly detection.

Bernoulli Naive Bayes

Bernoulli Naive Bayes is applicable for discrete data, specifically when features are binary. To extend from the multinomial example, instead of word counts, the data should be either 0s or 1s for if the word was included. The other applications from the multinomial applications would be eligible as well, as long as it is used in a binary sense.

Categorical Naive Bayes

Categorical Naive Bayes is "suitable for classification with discrete features that are categorically distributed" [sklearn documentation]. Note that this method requires encoding which represents the categorical variables as numerical. Ordinal encoding ensures order of the categorical variables (i.e. letter grades have an order), and non-ordinal categorical data (i.e. no implied order) can still be encoded. Some specific applications could be retail and marketing customer segmentation, healthcare classification, and others. Essentially, if categorical columns are present in the classification problem, this could be a decent method.

Analysis Code

The code for this analysis can be found [here].

Data Preparation

This project will be applying three of the Naive Bayes techniques:

• Multinomial Naive Bayes
• Gaussian Naive Bayes
• Bernoulli Naive Bayes

Each of the methods requires different data preparation, which will be detailed below.

Initial Data

Following from the previous sections, this section will use the same datasets. As a reminder, here is a preview of the data that will be used:

Ski Resorts Data

Weather Data

Google Places Data

Using these datasets, the following preparation will take place to prepare them for the different Naive Bayes methods.

Multinomial Naive Bayes

The most recent full year of data for weather data was from 2023, and the weather type occurences are currently in binary format for daily data. This data was summed for each resort for the 2023 year.

The ski resort data additionally has the number of trails per difficulty and the number lifts, which will be used in this analysis.

Using this count data, the type of Pass for resorts will attempted to be modeled. The possible options are:

• Ikon
• Epic
• Other

Below is a snippet of the prepared data for the Multinomial Naive Bayes data.

Gaussian Naive Bayes

Due to the machine learning method being able to continuous data, the entirety of the weather dataset was used for this Naive Bayes technique.

However, a small change was made to the label data for this dataset. To balance the label dataset, "partly-cloudy-day", "cloudy", "wind", and "fog" were transformed into "other".

This reduced the overall labels into:

• snow
• rain
• clear-day
• Other

Below is a snippet of the prepared data for the Gaussian Naive Bayes data.

Bernoulli Naive Bayes

For the Bernoulli setup, the data used was to test the efficacy of the Google Places API itself. The data was encoded to account for the return category calls for the API in 0s and 1s.

Using this binomial type data, the following label categories are:

• Medical
• Lodging
• Shopping
• Restaurants
• Grocery
• Spas
• Bars

Essentially, the Google API also returns a list of accompanying categories for the call category. This will test if there are any patterns between the call category and the other returns.

Below is a snippet of the prepared data for the Multinomial Naive Bayes data.

Training and Testing Sets

Additionally, training and testing sets were created. The two sets are disjoint, and must be disjoint. Using non-disjoint data between testing and training won't give an accurate representation of the performance of the model. First, this could result in an overfit of the model, which could end up describing noise, rather than the underlying distribution. Second, the testing set being non-disjoint helps to represent real-world data (i.e. unseen data).

Multinomial Naive Bayes Training Dataset

Multinomial Naive Bayes Testing Dataset

Gaussian Naive Bayes Training Dataset

Gaussian Naive Bayes Testing Dataset

Bernoulli Naive Bayes Training Dataset

Bernoulli Naive Bayes Testing Dataset