Coursera-Mathematics for Machine Learning: PCA Week4-1

Introduction

We have:

• a dataset $X$X in $R^D$RD consisting of n vectors (n training examples)

• n vectors $X_1$X1​, …, $X_n$Xn​ where $X_i$Xi​ are D-dimensional vectors

Objective:

• find a low dimensional representation of the data that is as similar to $X$X as possible.

Three important concepts

1. linear combination of the basis vectors

The first one is that every vector in $R^D$RD can be represented as a linear combination of the basis vectors.

$X_n$Xn​ can be written as the sum of i = 1 to D of $\beta_{in}$βin​ * $b_i$bi​.

• D个basis vectors $b_i$bi​的线性加和
• $b_i$bi​ are an orthonormal bases of $R^D$RD
• orthonormal: 正交（各自垂直）
  

2. orthogonal projection onto 1-D subspace

We can interpret $\beta_{in}$βin​ as the orthogonal projection of $X_n$Xn​ onto the one dimensional subspace spanned by the $i_{th}$ith​ basis vector.

3. orthogonal projection of X onto the subspace spanned by the M basis vectors

If we have an orthonormal basis $b_1$b1​ to $b_m$bm​ of $R^D$RD and we define $B$B to be the matrix that consists of these orthonormal basis vectors.

Then the projection of $X$X onto the subspace, we can write as $\widetilde X$X is $B * B.T * X$B∗B.T∗X.

That means $\widetilde X$X is the orthogonal projection of X onto the subspace spanned by the M basis vectors.

And $B.T * X$B.T∗X are the coordinates of $\widetilde X$X with respect of the basis vectors collected in the matrix $B$B. This is also called the code, so coordinates or code.

PCA

The key idea in PCA

To find a lower dimensional representation $\widetilde X_n$Xn​ that can be expressed using fewer basis vectors, let’s say M.

Assumptions:

• The data is centered, that means the dataset has mean zero.
• $b_1$b1​ to $b_D$bD​ are an orthonormal bases of $R^D$RD.

Generally, we can write any $\widetilde X_n$Xn​ in the following way:

This entire thing is still living in $R^D$RD. So, we took our general way of writing any vector in $R^D$RD which comes from property one, and we split the sum in property one into two sums. One is living in an M-dimensional subspace and the other one is living in a D minus M-dimensional subspace which is an orthogonal complement to this particular subspace.

In PCA, we ignore the second term, so we get rid of this part.

the principal subspace

And then we call the subspace that is spanned by the basis vectors $b_1$b1​ to $b_M$bM​ the principal subspace. So $b_1$b1​ to $b_M$bM​ span the principal subspace.

Although $\widetilde X_n$Xn​ is still a D-dimensional vector, it lives in an M-dimensional subspace of $R^D$RD and only M coordinates; $\beta_{n1}$βn1​ to $\beta_{nM}$βnM​ are necessary to represent it.

So, these ones are the coordinate of this, $\widetilde X_n$Xn​ vector. The betas of n also called the code of the coordinates of $\widetilde X_n$Xn​ with respect to the basis vectors $b_1$b1​ to $b_M$bM​.

Objective

And the setting now is as follows. Assuming we have data $X_1$X1​ to $X_n$Xn​, we want to find parameters $\beta_{in}$βin​ and orthonormal basis vectors $b_i$bi​, such that the average squared reconstruction era is minimised.

the average squared reconstruction error

J: the average squared reconstruction error

example

Let’s have a look at an example. We have data living in two dimensions and now we want to find a good one dimensional subspace such that the squared or average squared reconstruction error of the original data points and their corresponding projection is minimised.

Here I’m plotting the original dataset with their corresponding projections onto one dimensional subspaces and I’m cycling through a couple of options of subspaces and you can see that some of these projections are significantly more informative than others and in PCA we are going to find the best one. Our approach is to compute the partial derivatives of $J$J with respect to the parameters.

The parameters are the $\beta_{in}$βin​ and the $b_i$bi​.

We set the partial derivatives of J with respect to these parameters to zero and solve for the optimal parameters. But one observation we can already make. And that observation is that the parameters only enter this loss function through $\widetilde X_n$Xn​.

This means that in order to get our partial derivatives, we need to apply the chain rule. So, we can write $d_J$dJ​ by d either $\beta_{in}$βin​ or $b_i$bi​ can be written as $d_J$dJ​ by $d_{\widetilde X_n}$dXn​​ tilde times$d_{\widetilde X_n}$dXn​​ by d either $\beta_{in}$βin​ or $b_i$bi​.

We can already compute the first part.