互信息的原理、计算和应用
Background
熵 Entropy
在信息论中，熵是给定概率下的最佳编码方式[1]
交叉熵 Cross Entropy
条件熵 Conditional Entropy
KL-散度 KL-divergence
定义
计算方法
变分法 Variational approach
Mutual Information Neural Estimation, MINE
DEEP INFOMAX
应用
迁移学习
迁移强化学习
自监督学习
通过正负采样样本之间的互信息和图片的空间信息，网络在无监督的情况下学习图片深层的信息。
References
• [1] http://colah.github.io/posts/2015-09-Visual-Information/
• [2] Mutual Information Neural Estimation. https://arxiv.org/pdf/1801.04062.pdf
• [3] The IM Algorithm: A variational approach to Information Maximization. http://aivalley.com/Papers/MI_NIPS_final.pdf
• [4] https://zhuanlan.zhihu.com/p/39682125
• [5] https://arxiv.org/pdf/1801.04062.pdf
• [6] Learning Deep Representation By Mutual Information Estimation and Maximization. https://arxiv.org/pdf/1808.06670.pdf
• [7] Variational Information Distillation for Knowledge Transfer. https://openaccess.thecvf.com/content_CVPR_2019/papers/Ahn_Variational_Information_Distillation_for_Knowledge_Transfer_CVPR_2019_paper.pdf
• [8] Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch. https://arxiv.org/pdf/2006.07041.pdf