Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

• Loss Source 1: Cross entropy loss，各个阶段的分类器都有
• Loss Source 2: KL loss，深层的分类器作为浅层分类器的teacher
• Loss Source 3: L2 loss from hints，深层分类器的特征和浅层分类器的特征做L2 loss，bottleneck即feature adaptation，为了使student和teacher一样大