R：将“术语文档矩阵”转换为“语料库”

Question

我正在使用 R 编程语言。我正在尝试按照本教程的说明（https://cran.r-project.org/web/packages/tidytext/vignettes/tidying_casting.html）学习如何将“术语文档矩阵”转换为“语料库”。但是，我不清楚本教程中提供的解释，我不知道该怎么做。

使用公开的莎士比亚戏剧，我创建了术语文档矩阵，如下所示：

#load libraries
library(dplyr)
library(pdftools)
library(tidytext)
library(textrank)
library(tm)

#1st document
url <- "https://shakespeare.folger.edu/downloads/pdf/hamlet_PDF_FolgerShakespeare.pdf"

article <- pdf_text(url)
article_sentences <- tibble(text = article) %>%
  unnest_tokens(sentence, text, token = "sentences") %>%
  mutate(sentence_id = row_number()) %>%
  select(sentence_id, sentence)


article_words <- article_sentences %>%
  unnest_tokens(word, sentence)


article_words_1 <- article_words %>%
  anti_join(stop_words, by = "word")

#2nd document
url <- "https://shakespeare.folger.edu/downloads/pdf/macbeth_PDF_FolgerShakespeare.pdf"

article <- pdf_text(url)
article_sentences <- tibble(text = article) %>%
  unnest_tokens(sentence, text, token = "sentences") %>%
  mutate(sentence_id = row_number()) %>%
  select(sentence_id, sentence)


article_words <- article_sentences %>%
  unnest_tokens(word, sentence)


article_words_2<- article_words %>%
  anti_join(stop_words, by = "word")


#3rd document
url <- "https://shakespeare.folger.edu/downloads/pdf/othello_PDF_FolgerShakespeare.pdf"

article <- pdf_text(url)
article_sentences <- tibble(text = article) %>%
  unnest_tokens(sentence, text, token = "sentences") %>%
  mutate(sentence_id = row_number()) %>%
  select(sentence_id, sentence)


article_words <- article_sentences %>%
  unnest_tokens(word, sentence)


article_words_3 <- article_words %>%
  anti_join(stop_words, by = "word")

从这里，我创建了实际的“术语文档矩阵”：

library(tm)

#create term document matrix
tdm <- TermDocumentMatrix(Corpus(VectorSource(rbind(article_words_1, article_words_2, article_words_3))))

#inspect the "term document matrix" (I don't know why this is producing an error)
inspect(tdm)

现在，我不确定如何使用本教程 (https://cran.r-project.org/web/packages/tidytext/vignettes/tidying_casting.html) 中的说明并将“术语文档矩阵”转换为“语料库”。

这是应该的吗？

library(quanteda)
d <- quanteda::dfm(tdm, verbose = FALSE)

谁能告诉我如何解决这个问题？

谢谢

Answer 1

您可以尝试quanteda::corpus 将数据框转为语料库。

data <- rbind(article_words_1, article_words_2, article_words_3)
corp <- quanteda::corpus(data, text_field = 'word')

Answer 2

回答您在 Ronak 答案中的问题。

您无法将 tdm 转换为语料库，因为 tdm 已经在文档中汇总了字数，并且您丢失了句子的顺序。使用 quanteda，您可以在 dfm 上执行多项操作，例如替换单词或删除停用词。请参阅基于莎士比亚第一个文本的示例以显示该过程。

library(dplyr)
library(tidytext)
library(pdftools)

library(quanteda)

url <- "https://shakespeare.folger.edu/downloads/pdf/hamlet_PDF_FolgerShakespeare.pdf"

article <- pdf_text(url)
article_sentences <- tibble(text = article) %>%
  unnest_tokens(sentence, text, token = "sentences") %>%
  mutate(sentence_id = row_number()) %>%
  select(sentence_id, sentence)

# added count to simulate a dtm
article_words <- article_sentences %>%
  unnest_tokens(word, sentence) %>% 
  group_by(sentence_id, word) %>% 
  summarise(count = n())

# cast into a quanteda dfm
my_dfm <- cast_dfm(article_words, sentence_id, word, count)

# using quanteda's dfm_remove to remove stopwords from a dfm.
my_dfm <- dfm_remove(my_dfm, stopwords())

使用dfm_replace，您可以替换某些可能有错误和/或您不想使用的标点符号的单词。之后，您可以使用 dfm_compress 将具有相同名称的特征组合成 1 个特征。

但您最好尝试获取原始数据，而不是从 tdm 开始。