计算不在给定字典中的文本中的单词

Question

我如何查找和统计不在给定字典中的单词？

以下示例每次在文本中出现特定的字典单词（云和风暴）时都会计数。

library("quanteda")
txt <- "Forty-four Americans have now taken the presidential oath. The words have been spoken during rising tides of prosperity and the still waters of peace. Yet, every so often the oath is taken amidst gathering clouds and raging storms. At these moments, America has carried on not simply because of the skill or vision of those in high office, but because We the People have remained faithful to the ideals of our forbearers, and true to our founding documents."   
mydict <- dictionary(list(all_terms = c("clouds", "storms")))
dfmat <- tokens(txt) %>%
  tokens_select(mydict) %>%
  dfm()
dfmat

输出：

docs    clouds storms
  text1      1      1

我怎样才能生成字典中没有的所有单词的计数（云/风暴）？最好排除停用词。

例如，期望的输出：

docs    Forty-four Americans ...
  text1      1      1

Answer 1

当您检查 tokens_select 的帮助文件（运行 ?tokens_select）时，您可以看到第三个参数是 selection。默认值为"keep"，但您想要的是"remove"。由于这是很常见的事情，因此还有一个专用的tokens_remove 命令，我在下面使用它来删除停用词。

dfmat <- tokens(txt) %>%
  tokens_select(mydict, selection = "remove") %>%
  tokens_remove(stopwords::stopwords(language = "en")) %>% 
  dfm()
dfmat
#> Document-feature matrix of: 1 document, 38 features (0.00% sparse) and 0 docvars.
#>        features
#> docs    forty-four americans now taken presidential oath . words spoken rising
#>   text1          1         1   1     2            1    2 4     1      1      1
#> [ reached max_nfeat ... 28 more features ]

我认为这就是你想要做的。

^{由reprex package (v2.0.1) 于 2021 年 12 月 28 日创建}

Answer 2

这是一个使用 setdiff() 函数的例子。这是一个示例，说明如何从您的示例中提取 Obama 使用的词（以 $2013-Obama 为单位），而 Biden（以 $2021-Biden 为单位）未使用的词：

diff <- setdiff(toks[[1]], toks[[3]])