cs224n Lecture6 Language Models and RNNs

CS224n week4

• N-grams model
• PERPLEXITY'S RELATION TO ENTROPY

N-grams model

针对OOV两种方式

稀疏问题：
1.分子为0
使用Smoothing(discounting)

• Laplace smoothing(add-1 smoothing)：
  discount d_c
  
  

• Add-k smoothing：
  
  2.分母为0

• Backoff and Interpolation:
  we only “back off” to a lower-order n-gram if we have zero evidence for a higher-order n-gram
  held-out 、discount
  

• Katz backoff
  

• Kneser-Ney Smoothing

PERPLEXITY’S RELATION TO ENTROPY

• Entropy
  
• Entory rate