Lesson 118 — Principal Component Analysis (PCA)

Why is it necessary to center the data (subtract the mean) before applying PCA?

Given $$, compute the covariance and principal components.

Eigenvalues of $$: $$. Compute the explained variance by each PC and the cumulative. For which $$ does cumulative variance reach 90%?

PCs of a 2D dataset: $$, $$. Compute the scores of $$ in both components.

Using the data from the previous exercise, reconstruct $$ retaining only PC1. What is the reconstruction error?

Why is computing PCA via SVD of the data matrix $$ preferable to eigendecomposing $$ directly?

Compute the PCs of $$. What is the explained variance by each component?

With standardized data (z-score), what is the total variance? What does the Kaiser criterion of retaining only PCs with eigenvalue greater than 1 mean?

Dataset with eigenvalues $$ (total 10). Compute the mean squared reconstruction error when keeping K = 1 and K = 2 components.

PCA of 10 stocks' returns resulted in PC1 with similar magnitude positive loading for all stocks. What does PC1 represent economically? How would a portfolio manager use this information?

SVD of $$ with $$ samples gave singular values $$. Compute the corresponding covariance eigenvalues and the explained variance by PC1.

Prove that the scores of different principal components are uncorrelated with each other.

A dataset has 50 standardized features. With K = 10 PCs capturing 95% variance, how many parameters are needed to represent covariance via rank-K PCA versus full covariance?

Explain what the 3 first PCs of the Brazilian yield curve represent. Why do these 3 factors explain ~99% of variance?

Explain the difference between performing PCA with and without prior standardization (z-score). When should you NOT standardize?

Show that projection onto the K first PCs minimizes mean squared reconstruction error among all rank-K linear projections. What is the value of the minimum error in terms of eigenvalues?

Eigenvalues in descending order: 12, 8, 3, 1, 1, 1, 1, 1. Mentally construct the scree plot and identify the "elbow". How many PCs should you retain for 80% variance?

Describe the Eigenfaces method (Turk-Pentland 1991) for facial recognition using PCA. What dimensionality is achieved compared to original pixels?

What is the conceptual difference between PCA and ICA (Independent Component Analysis)? In what type of problem is ICA necessary?

What happens when the covariance matrix is the identity ($$)? What does this imply for PCA and dimensionality reduction?

Prove that eigenvectors of a symmetric matrix corresponding to distinct eigenvalues are orthogonal. Use this to justify the orthogonality of PCs.

Explain what a PCA biplot shows. How do you interpret the direction and length of feature arrows and the position of samples?

Why is classical PCA sensitive to outliers? What is the idea of Robust PCA for handling this problem?

Dataset: N = 1000 samples, d = 100 standardized features. PCA with K = 5 PCs explains 80% variance. Compute the data compression factor (ratio between original and PCA storage).

Explain the idea of Kernel PCA. How does replacing the inner product with a kernel allow capturing non-linear structure? What is the computational complexity?

Explain the "dual trick" of PCA: when $$ (more features than samples), how do you compute PCA efficiently? What is the complexity in each case?

PCA of 2023 ENEM microdata (5 grades: CN, CH, LC, MT, Essay) resulted in PC1 with similar magnitude positive loadings for all grades. Interpret PC1. What could PC2 represent?

Prove that the sample variance of the k-th score $$ equals the k-th eigenvalue of covariance $$. Use the connection with SVD.

In the 1000 Genomes Project, PCA of genomic data from ~2500 people across 26 populations reveals clusters by continent. Explain how this is possible and what the 3 first PCs represent genetically.

Describe the Probabilistic PCA model (Tipping-Bishop 1999). What are the advantages over classical PCA? How does this model reduce to classical PCA in a limiting case?