计算文本挖掘中的共现词网络

Question

我是 R 新手，我正在对多个 pdf 文件进行一些文本分析。 到目前为止，我已经设法绘制了 wordcloud 和条形图。但是，我还需要制作共现图或称为网络分析或链接图。任何人都可以帮助我指导如何将其应用于我当前的代码以及我应该使用什么库？ 代码如下：

require(pdftools)# reads pdf documents
require(tm)#text mining analysis
require(wordcloud)
require(RColorBrewer)

files<-list.files(pattern = "pdf$")#create vector of pdf file names (i included the pdf files in the same foldr)

alcohol<-lapply(files, pdf_text) #loads all the files

length(alcohol)# check the number of files

lapply(alcohol, length) #check the length of each file


pdfdatabase <- Corpus(URISource(files), readerControl = list(reader = readPDF)) #crearing a pdf database 
pdfdatabase <- tm_map(pdfdatabase, removeWords, stopwords("english")) 
pdfdatabase <- tm_map(pdfdatabase, removeNumbers) 
alcohol.tdm <- TermDocumentMatrix(pdfdatabase, control = list(removePunctuation = TRUE,
                                                              stopwords = TRUE,
                                                              tolower = TRUE,
                                                              streaming = FALSE,
                                                              removeNumbers = TRUE,
                                                              bounds = list(global = c(3, Inf))))



ft <- findFreqTerms(alcohol.tdm, lowfreq = 20, highfreq = Inf)

as.matrix(alcohol.tdm[ft,])

ft.tdm <- as.matrix(alcohol.tdm[ft,])
sort(apply(ft.tdm, 1, sum), decreasing = TRUE)



#find frequent terms
findFreqTerms(alcohol.tdm, lowfreq = 10)
#Examine frequent terms and their association
findAssocs(alcohol.tdm, terms = "sensor", corlimit = 0.5)




#convert term document matrix to data frame
m <- as.matrix(alcohol.tdm)
v <- sort(rowSums(m),decreasing = TRUE)
d <- data.frame(word = names(v), freq=v)


#create wrodcloud
set.seed(1234)
wordcloud(words = d$word, freq = d$freq, min.freq = 10,
          max.words = 200, random.order = FALSE, rot.per = 0.35,
          colors = brewer.pal(8, "Dark2"))



#Create Bar chart
barplot(d[1:11,]$freq, las = 2, names.arg = d[1:11,]$word,
        col = "lightblue", main = "Most frequent words",
        ylab = "Word freqencies")

提前致谢

a screen shot from the console after running the object ft.tdm

Answer 1

您可以从一个简单的卡方检验开始，以测试是否有大量文档同时具有这两个术语，而大量文档又缺少这两个术语。此测试可应用于所有术语对：

library(tidyverse)
library(broom)
library(ggraph)

# Ignore how many hits per document
ft.tdm.binary <-
  ft.tdm %>%
  as.data.frame() %>%
  mutate_all(~ ifelse(.x > 0, 1, 0)) %>%
  as.matrix()

co_occurrences <-
  ft.tdm %>%
  rownames() %>%
  combn(2) %>%
  t() %>%
  as_tibble() %>%
  rename(from = V1, to = V2) %>%
  mutate(
    test = list(from, to) %>% pmap(~ chisq.test(ft.tdm.binary[..1,], ft.tdm.binary[..2,]) %>% tidy())
  ) %>%
  unnest(test)

co_occurrences
#> # A tibble: 45 x 6
#>    from    to        statistic p.value parameter method                         
#>    <chr>   <chr>         <dbl>   <dbl>     <int> <chr>                          
#>  1 “alcoh… “just      2.05e-31   1             1 Pearson's Chi-squared test wit…
#>  2 “alcoh… “the       1.65e+ 0   0.199         1 Pearson's Chi-squared test wit…
#>  3 “alcoh… <U+FB01>…  1.23e-30   1.00          1 Pearson's Chi-squared test wit…
#>  4 “alcoh… <U+FB01>…  1.12e+ 0   0.290         1 Pearson's Chi-squared test wit…
#>  5 “alcoh… <U+FB01>…  4.47e- 1   0.504         1 Pearson's Chi-squared test wit…
#>  6 “alcoh… <U+FB01>…  2.07e- 1   0.649         1 Pearson's Chi-squared test wit…
#>  7 “alcoh… <U+FB01>…  4.66e- 4   0.983         1 Pearson's Chi-squared test wit…
#>  8 “alcoh… <U+FB01>…  1.56e- 1   0.692         1 Pearson's Chi-squared test wit…
#>  9 “alcoh… <U+FB01>…  4.47e- 1   0.504         1 Pearson's Chi-squared test wit…
#> 10 “just   “the       4.85e-30   1.00          1 Pearson's Chi-squared test wit…
#> # … with 35 more rows


co_occurrences %>%
  mutate(effect_size = sqrt(statistic / ncol(ft.tdm))) %>%
  # only show the significant co-occurrences
  filter(p.value < 0.05) %>%
  ggraph() +
    geom_node_label(aes(label = name)) +
    geom_edge_link(aes(color = effect_size))

请记住，如果有很多文档都缺少这两个术语，这实际上并没有多大意义。 对于罕见的单词尤其如此。 在这种情况下，可能需要改用 Jaccard 测试。

这是一种通用统计方法，不适用于文本文档的任何其他属性。