Quanteda：从 R 中的标记创建 ngrams 和 skipgrams

Question

我一直在浏览 R 中的 quanteda 包，但无法完全弄清楚 tokens_skipgrams 的功能。下面是the example from the manual of this package，我不太确定我是否理解得很好：

tokens_skipgrams(toks, n = 3, skip = 0:2, concatenator = " ")   
tokens from 1 document.
text1 :
[1] "insurgents killed in"        "insurgents killed ongoing"  
[3] "insurgents killed fighting"  "insurgents in ongoing"      
[5] "insurgents in fighting"      "insurgents ongoing fighting"
[7] "killed in ongoing"           "killed in fighting"         
[9] "killed ongoing fighting"     "in ongoing fighting"        

我希望输出包含以下内容：

 "insurgents killed in"    "killed in ongoing"    "in ongoing fighting" 
 "insurgents in fighting"

为什么结果包括：

  "insurgents killed ongoing"  
  "insurgents killed fighting"  
  "insurgents in ongoing"      
  "insurgents ongoing fighting"
  "killed in fighting"         
  "killed ongoing fighting" 

在上面的例子中，skip = 0:2 即 skip 是 0、1 和 2。因此，我认为上面的命令可以安全地分成 3 个部分，每个部分的组合会给我上面的结果正如我所指出的，我无法得到。

tokens_skipgrams(toks, n = 3, skip = 0, concatenator = " ")   
tokens from 1 document.
text1 :
[1] "insurgents killed in" "killed in ongoing"    "in ongoing fighting" 

tokens_skipgrams(toks, n = 3, skip = 1, concatenator = " ")   
tokens from 1 document.
text1 :
[1] "insurgents in fighting"


tokens_skipgrams(toks, n = 3, skip = 2, concatenator = " ")   
tokens from 1 document.
text1 :
character(0)

但结果的组合给了我我所期望的结果，而不是上面给出的那个。

有人可以帮我解决这个问题吗？

Answer 1

您观察到的行为是 Guthrie 等人 (2006) 对skiagram 定义的实现：“k skip-gram 是一个 ngram，它是所有 ngram 的超集，每个(k-i) skipgram 直到 (k-i)==0（包括 0 个 skip-gram）。” （这在?tokens_skipgram 的 quanteda 手册页上被引用。原始来源是 Guthrie, D., B. Allison, W. Liu, and L. Guthrie. 2006. "A Closer Look at Skip-Gram Modelling.".)。下面s02 的示例直接取自那篇论文，称为“2-skip-tri-grams”。

但是，对于skip 的标量值，没有实现跳过的这种递归实现，以便为用户提供最大的控制权。

这解释了在提供skip 值作为如上所述的单独比例，然后作为序列0:2 的区别。对于

toks <- tokens("insurgents killed in ongoing fighting")
toks
# tokens from 1 document.
# text1 :
# [1] "insurgents" "killed"     "in"         "ongoing"    "fighting" 

我们在skip = 0:2 时观察到诸如“叛乱分子在战斗中被杀”之类的组合，因为这包括 0（在“叛乱分子”和“被杀”之间）和 2（在“被杀”和“战斗”之间）。对于这里的短语，这意味着从 skip = 0:1 到 skip = 0:2 之间只有两个额外的跳过图：

(s01 <- tokens_skipgrams(toks, n = 3, skip = 0:1, concatenator = " "))
# tokens from 1 document.
# text1 :
# [1] "insurgents killed in"      "insurgents killed ongoing" "insurgents in ongoing"    
# [4] "insurgents in fighting"    "killed in ongoing"         "killed in fighting"       
# [7] "killed ongoing fighting"   "in ongoing fighting"      

(s02 <- tokens_skipgrams(toks, n = 3, skip = 0:2, concatenator = " "))
# tokens from 1 document.
# text1 :
# [1] "insurgents killed in"        "insurgents killed ongoing"   "insurgents killed fighting" 
# [4] "insurgents in ongoing"       "insurgents in fighting"      "insurgents ongoing fighting"
# [7] "killed in ongoing"           "killed in fighting"          "killed ongoing fighting"    
# [10] "in ongoing fighting"        

setdiff(as.character(s02), as.character(s01))
# [1] "insurgents killed fighting"  "insurgents ongoing fighting"