Modeling - PRincipal Component Analysis (PCA)

What is PCA?

Principal Component Analysis, or simply PCA, is a dimenson reduction technique which operates by consolidating information from multiple features into a new projection space in which each new feature is orthogonal to each other new feature. Specifically, PCA is a constrained optimization technique in which an eigenspace transformation is used to put quantitative data into a different orthonormal basis. PCA initialization requires standardization (which limits the variation of the data), from which a covariance matrix is computed. The eigenspace transformation extracts eigenvalues and eigenvectors from the covariance matrix of the standardized data. The covariance matrix step is crucial, as antyhing above a zero shows correlation, which is what needs to removed. In essence, the eigenvectors form the orthonormal basis due to them being uncorrelated. The associated eigenvalues are the explained information (or explained variance) of the eigenvectors. In a dataset which has no correlation between its variables, the eigenvectors would essentially be its columns and removing dimensions removes actual information. However, this is rare in real datasets. Furthermore, the covariance matrix is symmetric, which allows for the guaranteed existence of an orthonormal basis of corresponding vector space consisting of eigenvectors and corresponding real-valued eigenvalues.

What Data Can Be Used?

PCA is used on quantitative data. This analysis focuses on the quantitative data of the main datasets used throughout this project:

• Ski Resorts Data - Cleaning and Aggregation Process Described Here
• Weather Data - Cleaning and Aggregation Process Described Here
• Google Places Data - Cleaning and Aggregation Process Described Here

Ski Resorts Data - Snippet

Weather Data - Snippet

Google Places Data - Snippet

Data Preparation

Each dataset required some alteration in preparation for PCA. Namely, this included subsetting the data to quantitative values and separating the labels. The labels would be saved for later to compare with the results. Some of the datasets had multiple categorical data features which could be used as labels depending on the purpose of the analysis. Other columns were simply dropped. Thus, a concise script with which could perform this cleaning, along with applying the PCA algorithm and analysis of the results was created. This script can be found here, and contains detailed documentation on these functions.

Ski Resort Data - Preparation

Weather Data - Preparation

Google Places Data - Preparation

Applying Principal Component Analysis in Python

PCA in Python can be accomplished through the Scikit-Learn module, sklearn.decomposition.PCA. However, it is important to first normalize the quantitative data. Results can be skewed when values are significantly different between features. In other words, when features have much larger and smaller values than each other. To accomplish normalization, another Scikit-Learn module was used, sklearn.preprocessing.StandardScaler. Each feature has its mean removed and is scaled to unit variance.

PCA can be applied in a generic sense, without specifying how many principal components are to be returned. This will create a return of as many principal components as there are input features. PCA can also be applied with a desired number of components to be returned. One point of confusion with either of these methods is how the original features relate to the output. It's important to understand that PCA transforms or projects the data into a different space using eigenvalues and eigenvectors. There isn't exactly a one-to-one relationship between the projected data onto principal components and the features of the original dataset. To further illustrate PCA, analyze its results, and try to make sense of a relationship between original features and the PCA projection, several attributes of sklearn's PCA model will be used. Given a model was created with the following code:

    
        # sklearn libraries
        from sklearn.decomposition import PCA
        from sklearn.preprocessing import StandardScaler
        
        # normalize pandas dataframe with only quantitative features
        scaler = StandardScaler()
        df_normal = scaler.fit_transform(df)
        
        # create the pca model and project data into PCA space
        pca = PCA()
        pca_projection = pca.fit_transform(df_normal)
        
        # obtain eigenvalues
        eigenvalues = pca.explained_variance_
        
        # explained variance
        eigenvalue_ratios = pca.explained_variance_ratio_
        
        # obtain eigenvectors
        eigenvectors = pca.components_
        
        # obtain loadings matrix
        loadings_matrix = pd.DataFrame(pca.components_.T, columns=[f'principal_component_{col+1}' for col in range(pca.components_.shape[0])], index=df.columns)
    

• eigenvalues: the amount of variance explained by each of the selected components
• eigenvectors: directions of maximum variance in the data, sorted by decreasing eigenvalues
• eigenvalue ratios: percentage of variance explained by each of the selected components
• loadings matrix: represents the correlation between original features and principal components
• pca projection: original data projected onto the pca space

PCA and Analysis Process

Each dataset was processed and the results were analyzed in this script, for full-feature PCA, 3-dimensional PCA, and 2-dimensional PCA via the following:

1. PCA Application and Key Attribute Extraction perform_pca()
2. Orthogonality Validated validate_orthogonality()
3. Visualize Variance visualize_variance()
• Full Dimensional PCA Includes Additional Outputs for Components Required for Retention of 95% Explained Variance
4. Explained Variance
• Full Dimensional PCA Includes Additional Output for the Top 3 Eigenvalues
5. Loadings Matrix Analysis
• Loadings Matrix Barplot
• Loadings Matrix Boxplot
6. Further Visualizations
• 3-Dimensional PCA with Labeled Hue - Weather Includes an Animated Time Series Visualization
• 2-Dimensional PCA with Labeled Hue

The Loadings Matrix

Loadings Matrices represent the correlation between the original variables and the principal components. When PCA is performed, the new principal components are a consolidation of information of the original variables. Therefore, each principal component could be influenced by each of the original variables (i.e. potentially contain information from each original variable). The loadings matrix shows this influence (direction and strength) amount by calculating correlations. Closer to zero, the less influence. A positive correlation indicates that higher scores on the factor are associated with higher scores on the variable. A negative correlation indicates that higher scores on the factor are associated with lower scores on the variable. Higher negative correlations (in an absolute sense) are indicative of high influence, just inversely!

Essentially, these correlations help in understanding which factors influence which variables and whether this influence is direct or inverse. By analyzing loadings matrices, the true power of PCA and its consolidation properties are revealed. To further illustrate this property, it can be beneficial to investigate the absolute values of a loadings matrix. Using absolute values, the correlations will be investigated via the following:

• Correlation Ranking
1. Turn correlations into their absolute value correlations.
2. Obtain rank of each original variable across each principal component. The lower the rank, the higher the absolute correlation, the more influenced a principal component is by the original variable.
3. Analyze the rankings
• Average Correlation Ranking: a low average absolute correlation rank indicates this original variable has greater influence across the principal components, thus a more substantial amount of information is obtained from that original variable.
• Correlation Ranking Spread: a mean is a common statistical measure, but the overall spread of rankings can illustrate further statistical measures and influence as a whole.

PCA Results and Analysis

Summary of PCA Results and Analysis

Principal Component Analysis was applied to three main datasets relevant to this topic. Full Feature PCA, Three Dimensional PCA, and Two Dimensional PCA results were analyzed. Specifically, eigenvectors and eigenvalues from the data projected into PCA spaces were investigated, with emphasis on how much information was retained by the PCA process. An additional component of the analysis used loadings matrices in an attempt to understand the strength and direction each original feature had on the principal componets (new features). Illustrations of the projected data were made for three dimensionsal PCA and two dimensional PCA, with labels applied to help detect potential patterns.

Some interesting takeaways:

• Loadings matrices and subsequently the influence of the original features on the principal components changed with reduced principal components, namely in the datasets with a greater number of features.
• Visualizations on the resort data with different labels did reveal some relevant cluster patterns.
• Visualizations on the weather data also revealed some relevant cluster patterns, and was the most symmetric of the three datasets when projected into the PCA space.
• Visualizations on the google places data did have some relevant cluster patterns revealed, but the data projected into the PCA space produced an overall shape much different to the other datasets. Even though outliers were addressed while cleaning this data, could hidden outliers be responsible for the uniqueness of the PCA projection results?