信息科学原理第一章(香农熵，条件熵，相对熵)

@(信息科学原理)

• 导论
  • 香农熵
    • 联合熵
  • 互信息
  • 条件熵
  • 相对熵(KL-散度)
  • 交叉熵
  • 边缘概率，条件概率，联合概率
  • - 条件概率计算的是P(y|x)=P(x,y)P(x)$P(y|x)=\frac{P(x,y)}{P(x)}$
    • example

导论

香农熵

信息：h(x)=−logp(x)$h(x)=-\log p(x)$

H(X,Y)=−∑x∈XP(x)log(P(x))=Ex∼Plog(P(x))$$\begin{array}{r}H(X,Y)=-\sum _{x\in X}P(x)\log (P(x))\\ =E_{x\sim P}log(P(x))\end{array}$$

其中0log0=0$0\log 0=0$，并且定义log1e=1nats$\log \frac{1}{e}=1nats$和 log12=1bits$\log \frac{1}{2}=1bits$

联合熵

H(X,Y)=−∑x∈X,y∈YP(x,y)logP(x,y)=Ex∼PlogP(x,y)$$\begin{array}{l}H(X,Y)=-\sum _{x\in X,y\in Y}P(x,y)\log P(x,y)\\ =E_{x\sim P}\log P(x,y)\end{array}$$

互信息

I(X,Y)=∑x∈X,y∈YP(x,y)logP(x,y)P(X)P(Y)=Ex,y∼PlogP(x,y)P(X)P(Y)=DKL(P(x,y)∣∣P(X)P(Y))$$\begin{array}{l}I(X,Y)=\sum _{x\in X,y\in Y}P(x,y)\log \frac{P(x,y)}{P(X)P(Y)}\\ =E_{x,y\sim P}\log \frac{P(x,y)}{P(X)P(Y)}\\ =D_{KL}\left(P(x,y)\mid \mid P(X)P(Y)\right)\end{array}$$

衡量两个信息的相关性大小的量

条件熵

H(Y|X)=−∑x∈X,y∈YP(x,y)logP(y|x)=−∑x∈X,y∈YP(x,y)logP(x,y)P(x)=∑x∈X,y∈YP(x,y)logP(x)P(x,y)=Ex,y∼PlogP(x)P(x,y)$$\begin{array}{l}H(Y|X)=-\sum _{x\in X,y\in Y}P(x,y)\log P(y|x)\\ =-\sum _{x\in X,y\in Y}P(x,y)\log \frac{P(x,y)}{P(x)}\\ =\sum _{x\in X,y\in Y}P(x,y)\log \frac{P(x)}{P(x,y)}\\ =E_{x,y\sim P}log\frac{P(x)}{P(x,y)}\end{array}$$

知道的信息越多，随机事件的不确定性就越小

proof: H(X,Y)=H(X)+H(Y|X)$H(X,Y)=H(X)+H(Y|X)$ :

H(X,Y)=−∑x∈X,y∈YP(x,y)logP(x,y)=−∑x∈X,y∈YP(x,y)log[P(y|x)P(x)]=−∑x∈X,y∈YP(x,y)[logP(y|x)+logP(x)]=−∑x∈X,y∈YP(x,y)logP(y|x)+[−∑x∈XP(x)logP(x)]=H(Y|X)+H(x)$$\begin{array}{l}H(X,Y)=-\sum _{x\in X,y\in Y}P(x,y)\log P(x,y)\\ =-\sum _{x\in X,y\in Y}P(x,y)\log \left[P(y|x)P(x)\right]\\ =-\sum _{x\in X,y\in Y}P(x,y)\left[\log P(y|x)+\log P(x)\right]\\ =-\sum _{x\in X,y\in Y}P(x,y)\log P(y|x)+[-\sum _{x\in X}P(x)\log P(x)]\\ =H(Y|X)+H(x)\end{array}$$

proof:H(X,Y|Z)=H(X|Z)+H(Y|X,Z)$H(X,Y|Z)=H(X|Z)+H(Y|X,Z)$

H(X,Y|Z)=−∑x,y,zP(x,y,z)logP(x,y|z)=−∑x,y,zP(x,y,z)log[P(x,y,z)P(z)]=−∑x,y,zP(x,y,z)log[P(x,y,z)P(x,z)P(x,z)p(z)]=[−∑x,y,zP(x,y,z)logP(x,y,z)P(x,z)]+[−∑x,y,zP(x,y,z)logP(x,z)P(z)]=[−∑x,y,zP(x,y,z)logP(x,y,z)P(x,z)]+[−∑x,zP(x,z)logP(x,z)P(z)]=H(Y|X,Z)+H(X|Z)$$\begin{array}{l}H(X,Y|Z)=-\sum _{x,y,z}P(x,y,z)\log P(x,y|z)\\ =-\sum _{x,y,z}P(x,y,z)\log \left[\frac{P(x,y,z)}{P(z)}\right]\\ =-\sum _{x,y,z}P(x,y,z)\log \left[\frac{P(x,y,z)}{P(x,z)}\frac{P(x,z)}{p(z)}\right]\\ =\left[-\sum _{x,y,z}P(x,y,z)\log \frac{P(x,y,z)}{P(x,z)}\right]+\left[-\sum _{x,y,z}P(x,y,z)\log \frac{P(x,z)}{P(z)}\right]\\ =\left[-\sum _{x,y,z}P(x,y,z)\log \frac{P(x,y,z)}{P(x,z)}\right]+\left[-\sum _{x,z}P(x,z)\log \frac{P(x,z)}{P(z)}\right]\\ =H(Y|X,Z)+H(X|Z)\end{array}$$

相对熵(KL-散度)

DKL(P∣∣Q)=∑x∈XP(x)logP(x)Q(x)=Ex∼P[logP(x)Q(x)]=Ex∼P[logP(x)−logQ(x)]$$\begin{array}{l}{D}_{KL}(P\mid \mid Q)\\ =\sum _{x\in X}P(x)\log \frac{P(x)}{Q(x)}\\ =E_{x\sim P}\left[\log \frac{P(x)}{Q(x)}\right]\\ =E_{x\sim P}\left[\log P(x)-\log Q(x)\right]\end{array}$$

note:DKL(P∣∣Q)≥0$D_{KL}(P\mid \mid Q)\ge 0$,用于衡量两个分布的相似性

交叉熵

H(P,Q)=H(P)+DKL(P∣∣Q)H(P,Q)=−Ex∼PlogQ(x)$$\begin{array}{l}H(P,Q)=H(P)+D_{KL}(P\mid \mid Q)\\ H(P,Q)=-E_{x\sim P}\log Q(x)\end{array}$$

边缘概率，条件概率，联合概率

- 边缘概率就是计算每一边
- 联合概率计算的是P(X=x,Y=y)=P(y|x)P(x)$P(X=x,Y=y)=P(y|x)P(x)$

- 条件概率计算的是P(y|x)=P(x,y)P(x)$P(y|x)=\frac{P(x,y)}{P(x)}$

对于离散的随机变量：

对于连续的随机变量：

example

H(X)=−∑x∈XP(x)logp(x)=12log2+14log4+18log8+18log8=74log2=74bits$$\begin{array}{l}H(X)=-\sum _{x\in X}P(x)\log p(x)\\ =\frac{1}{2}\log 2+\frac{1}{4}\log 4+\frac{1}{8}\log 8+\frac{1}{8}\log 8\\ =\frac{7}{4}\log 2=\frac{7}{4}bits\end{array}$$

H(X|Y)=−∑x∈Xy∈YP(x,y)logP(x,y)P(y)=432log1/44/32+232log1/42/32+232log1/42/32+⋅⋅⋅=118bits$$\begin{array}{l}H(X|Y)=-\sum _{x\in Xy\in Y}P(x,y)log\frac{P(x,y)}{P(y)}\\ =\frac{4}{32}\log \frac{1/4}{4/32}+\frac{2}{32}\log \frac{1/4}{2/32}+\frac{2}{32}\log \frac{1/4}{2/32}+\cdot \cdot \cdot \\ =\frac{11}{8}bits\end{array}$$

H(X,Y)=−∑x∈Xy∈YP(x,y)logP(x,y)=278bits$$\begin{array}{l}H(X,Y)=-\sum _{x\in Xy\in Y}P(x,y)logP(x,y)\\ =\frac{27}{8}bits\end{array}$$