Modeling - Support Vector Machines

Overview of SVM

Support Vector Machines (SVMs) are supervised learning methods which transform features into a higher dimensional space to separate the labels. The usefulness of an SVM comes from when the input data in its original dimensional space isn’t linearly separable, but in a higher dimensional space there exists a hyperplane which can linearly separate the groups of the data.

In the example above, the groups in the data are not linearly separable in their original two-dimensional space, however, transformed into a three-dimensional space, a three-dimensional plane is able to linearly separate the data. A hyperplane which exists in 4 or more dimensions, although it cannot be visualized, can be conceptualized mathematically and theoretically.

SVMs use a quadratic optimization algorithm, in which the final optimal dual form contains a dot product (or inner product). This allows for the use of kernels, which are functions that return an inner product in a higher dimensional space. Being able to apply kernels is essential, as just the solution to the dot product is needed and doesn’t actually need to be transformed into a higher dimensional space in practice. The example above takes a small amount of data in 2-dimensions and transforms them into 3-dimensions. However, if millions of points of data are transformed into a dimensional space in the thousands (or even into an infinite dimensional space), the problem becomes intractable. To reiterate, being able to use a dot product, and subsequently a kernel, allows for just the solution of the dot product to be used instead of actually transforming the data into a higher dimensional space. This makes SVMs highly efficient. Additionally, SVMs create a margin between the groups in the higher dimensional space. Any point on the margin is known as a support vector. Not only are they computationally efficient but they are also more resistant to outliers and noise due to this. Keep in mind that a single SVM is a binary classifier, however multiple SVMs can be ensembled together for more than a 2-class problem

Ommitting the initial mathematics to obtain the final optimal dual form, the main equation becomes:

In otherwords, solve for:
With the following constraints:
Maximizing and Minimizing these constraints now becomes an optimization problem:
Finally, substituting these optimal results into L gives:
Where:

• The \( y_i \)'s are the label or group of data (usually represented by \( +1 \) and \( -1 \))
• The \( \lambda_i \)'s are the Lagrange Multipliers
• The \( x_i \)'s are the data vectors, and \( x_i^T x_j \) is the dot product
• The \( \lambda_i \)'s and \( y_i \)'s are scalars, while the \( x_i \)'s are vectors of data in their original dimensional space

There are several common kernels used with the SVM technique. Essentially, if a potential fucntion for SVM can be written as an inner product, then it can be used as a kernel in SVM. This section will talk about the Polynomial Kernel and the Radial Basis Function (RBF) Kernel.

• Polynomial Kernel
• Function: \( (a^Tb + r)^d \)
• \( a \text{ and } b \) are vectors
• \( r \) is the coefficient of the polynomial, which helps to control the size of the margin
• \( d \) is the degree of the polynomial, which helps control the comlexity of the model
• Idea: Uses a polynomial function to map the data into a higher-dimensional space by taking the dot product of the data in the original dimensional space and mapping this to the higher dimensional space with the polynomial function
• RBF Kernel
• Function: \( e^{-\gamma (a - b)^2} \)
• \( a \text{ and } b \) are vectors
• \( \gamma \) scales the result, and represents \( \frac{1}{2 \sigma^2} \)
• By substituting \( \gamma \) back in, the normal distribution formula is created, thus the RBF is based on the normal distribution
• Idea: Using the Gaussian function, the RBF kernel maps the data into an infinite-dimensional space
• Even a function which maps data into an infinite space can be shown to be a kernel (i.e. written as a dot product)

Simple Example With the Polynomial Kernel

Given a Polynomial Kernel with parameters \( r = 1, d = 2 \) on data in an original 2-dimensional dataset:

This can be shown to a be a dot product which can "cast" points into the proper number of dimensions:

This can be written as a dot product of two transformed points, \( transform_{1} \cdot transform_{2} \):

Thus, 2-dimensional data is "cast" or "projected" into a 6-dimensional space.
Applying this to an example,

Finally, using a point from this data, \( a = [a_1, a_2] = [1.5, 2] \):

Data Preparation

To preface the data used in this, SVMs can only work on labeled numeric data. First, an SVM is a supervised machine learning method. This means, that it can only be used on labeled data in order to train the model. Second, due to the mathematic nature of dot products, and subsequently kernels, the data must be numeric.
The data used for this model will be the weather data featured in many of the models throughout this project. The goal will be to determine the icon used by a weather system on a given day of data. Options will be:

• Clear Day
• Rain
• Snow
• Other

The icon label was highly disproportionate. Wind and Fog categories were placede into the Other category. After that step, the samples were downsized to be proportionate to match the lowest category. Overall, there were still over 500,000 datapoints to train the data on.
Note that SVMs are binary classifiers. When a multi-class problem is presented, ensemble learning must be used to link several SVM models together. However, libraries like Scikit-Learn automatically ensemble.
Certain numeric variables directly associated with Rain, Snow, and Wind were disregarded as these are essentially what is trying to be predicted in the categorical label of icon. The variables used in the analysis were:

• Maximum Temperature
• Minimum Temperature
• Temperature
• Fells Like Maximum (temperature)
• Fells Like Minimum (temperature)
• Feels Like (temperature)
• Dew Point
• Humidity
• Pressure
• Day of the Year

Using these numeric variables, principal component analysis (PCA) was performed to reduce the dimensionality of the data while retaining as much information as possible.





Utilizing PCA retains over 80% of information in the first two principal components, while over 90% is retained within the first three principal components. With over 80% of information retained with the first two principal components, these will be the data columns used. The upside to this is that the data will be known to begin in two-dimensions and trasnformed further.
Additionally, training and testing sets were created. The two sets are disjoint, and must be disjoint. Using non-disjoint data between testing and training won't give an accurate representation of the performance of the model. First, this could result in an overfit of the model, which could end up describing noise, rather than the underlying distribution. Second, the testing set being non-disjoint helps to represent real-world data (i.e. unseen data).
The explained processes above can be examined in the datasets below.

Weather Data

Data Prepared for SVM

Training Data for SVM

Testing Data for SVM

Coding SVM

The code for the data preparation and performing SVM can be found [here].
The kernels mentioned in the overview, Polynomial and RBF will be used. Additionally, a third kernel known as the Sigmoid Kernel will be used as well. Different cost parameters will be utilized to find the best model.
Please note that the data was trained and tested on a subset of 1% of the original data. This still resulting in thousands of rows to train the models on.

Polynomial Kernel

For the Polynomial Kernel, the kernel with the following parameters performed the best:

• Degree: 3
• C: 2.0
The confusion matrices and results are listed below.

RBF Kernel

For the RBF, altering C only seemed to slightly change the confusion matrix while the overall accuracy remained the same. Therefore, it makes sense to leave the C value at the default of 1.

• C: 1.0
The confusion matrices and results are listed below.

Sigmoid Kernel

For the Sigmoid, increasing C seemed to decrease the accuracy. Thus a final C value was tested in the opposite direction to obtain optimal results.

• C: 0.5
The confusion matrices and results are listed below.

SVM Results

Although not extremely desirable results, the RBF with the default cost parameter of C = 1.0 performed the best overall. To illustrate these results, a larger subset of 5% of the data on 3 principal components was trained. This resulted in an accuracy of 60.92%.

SVM Conclusions

An analysis was performed to examine if a better method to categorize the weather for a given day existed. The possible categories that could result were Clear, Rain, Snow, or Other. Overall, the categories of Clear, Rain, and Snow were able to be predicted with decent accuracy. However, the category of Other was more difficult to predict given the models provided. Interestingly enough, when predicting between Snow and Rain, there is a less of chance to falsely predict this. In other words, on days when it would snow or rain, if they were incorrectly predicted, out of the potential categories, it's more likely to be predicted as either Clear or Other. Other contains phenomena such as fog, wind, and overcast.