Correlation between variables in a -dimensional dataset are often
summarized by a
covariance matrix.
To get a better understanding of how correlation matrices characterize correlations between data points, we plot data points drawn from 3 different 2-dimensional Gaussian distributions, each of which is defined by a different covariance matrix.
The left plots below display the covariance
matrix for each Gaussian distribution. The values along the diagonal represent the variance of the data along each dimension, and the off-diagonal values represent the covariances between the dimensions. Thus the
-th
entry of each matrix represents the correlation between the
-th
and
-th dimensions. The right plots show data drawn from the
corresponding 2D Gaussian.
The top row plot display a covariance matrix equal to the identity matrix, and the points drawn from the corresponding Gaussian distribution. The diagonal values are 1, indicating the data have variance of 1 along both of the dimensions. Additionally, the off-diagonal elements are zero, meaning that the two dimensions are uncorrelated. We can see this in the data drawn from the distribution as well. The data are distributed in a sphere about origin. For such a distribution of points, it is difficult (impossible) to draw any single regression line that can predict the second dimension from the first, and vice versa. Thus an identity covariance matrix is equivalent to having independent dimensions, each of which has unit (i.e. 1) variance. Such a dataset is often called “white” (this naming convention comes from the notion that white noise signals–which can be sampled from independent Gaussian distributions–have equal power at all frequencies in the Fourier domain).
The middle row plots the points that result from a diagonal, but not identity covariance matrix. The off-diagonal elements are still zero, indicating that the dimensions are uncorrelated. However, the variances along each dimension are not equal to one, and
are not equal. This is demonstrated by the elongated distribution in red. The elongation is along the second dimension, as indicated by the larger value in the bottom-right (point )
of the covariance matrix.
The bottom row plots points that result from a non-diagonal covariance matrix. Here the off-diagonal elements of covariance matrix have non-zero values, indicating a correlation between the dimensions. This correlation is reflected in the distribution of drawn datapoints (in blue). We can see that the primary axis along which the points are distributed is not along either of the dimensions, but a linear combination of the dimensions.
The MATLAB code to create the above plots is here
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
|
%
INITIALIZE SOME CONSTANTS
mu
= [0 0]; %
ZERO MEAN
S
= [1 .9; .9 3]; %
NON-DIAGONAL COV.
SDiag
= [1 0; 0 3]; %
DIAGONAL COV.
SId
= eye(2); %
IDENTITY COV.
%
SAMPLE SOME DATAPOINTS
nSamples
= 1000;
samples
= mvnrnd(mu,S,nSamples)';
samplesId
= mvnrnd(mu,SId,nSamples)';
samplesDiag
= mvnrnd(mu,SDiag,nSamples)';
%
DISPLAY
subplot(321);
imagesc(SId);
axis image,
caxis([0
1]), colormap hot, colorbar
title('Identity
Covariance')
subplot(322)
plot(samplesId(1,:),samplesId(2,:),'ko');
axis square
xlim([-5
5]), ylim([-5 5])
grid
title('White
Data')
subplot(323);
imagesc(SDiag);
axis image,
caxis([0
3]), colormap hot, colorbar
title('Diagonal
Covariance')
subplot(324)
plot(samplesDiag(1,:),samplesDiag(2,:),'r.');
axis square
xlim([-5
5]), ylim([-5 5])
grid
title('Uncorrelated
Data')
subplot(325);
imagesc(S);
axis image,
caxis([0
3]), colormap hot, colorbar
title('Non-diagonal
Covariance')
subplot(326)
plot(samples(1,:),samples(2,:),'b.');
axis square
xlim([-5
5]), ylim([-5 5])
grid
title('Correlated
Data')
|