如何在 R 文本挖掘中更改 termDocumentmatrix 的语言？

Question

我需要在 termDocumentmatrix 的功能中将语言更改为土耳其语。你能帮我吗？

此代码有效。我得到了我想要的土耳其语词干、停用词等结果。

dat<-"BirGün, Türkiye'de günlük olarak yayımlanan ulusal bir gazete.
Gazetenin yazı işleri müdürü Berkant Gültekin, yayın danışmanı Barış İnce, 
sorumlu müdürü Cansever Uğur ve haber koordinatörü İbrahim Varlı'dır. Yayın 
hayatına 14 Nisan 2004'te başlayan gazetenin sahibi Birgün Yayıncılık ve İletişim Ticaret AŞ'd"

dat%>% 
  tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE)%>%
  tokens_remove(stopwords("tr", source = "stopwords-iso")) %>%
  tokens_wordstem(language = "turkish") %>%
  tokens_tolower() 

Result:

[1] "birg"        "türkiye'"    "günlük"      "yayımlana"   "ulusal"      "gaze"        "gazete"      "yaz"         "iş"         
[10] "müdür"       "berkant"     "gültek"      "yay"         "danışma"     "barış"       "ince"        "sorumlu"     "müdür"      
[19] "cansever"    "uğur"        "haber"       "koordinatör" "ibrah"       "varlı'"      "yay"         "hayat"       "nisa"       
[28] "te"          "başlaya"     "gazete"      "sahip"       "birgi"       "yayıncılık"  "iletiş"      "ticaret"     "aş'd"       

但是，我没有将这些过程整合到术语文档矩阵中，我尝试挖掘 pdf 文件

library(pdftools)
library(tm)
library(SnowballC)
library(dplyr)
library(stringr)
library(tidytext)
library(quanteda)

这部分是在工作区读取pdf文件

files <- list.files(pattern = "pdf$")
file<-as.character(files)
opinions <- lapply(files, pdf_text)
length(opinions)
lapply(opinions, length) 

创建语料库

corp <- Corpus(URISource(files),
               readerControl = list(reader = readPDF))

在这部分，"language="turkish" 不起作用，它仍然使用英语作为基础语言

opinions.tdm <- TermDocumentMatrix(corp, 
                                   control = 
                                     list(language="turkish",
                                          stopwords = TRUE,
                                          removePunctuation = TRUE,
                                          tolower = TRUE,
                                          stemming = TRUE,
                                          removeNumbers = TRUE,
                                          bounds = list(global = c(1, Inf)))) 

inspect(opinions.tdm[1:10,]) 

opinions.tdm <- TermDocumentMatrix(corp, 
                                   control = 
                                     list(language="turkish",
                                          stopwords = TRUE,
                                          tolower = TRUE,
                                          stemming = TRUE,
                                          removePunctuation=TRUE,
                                          removeNumbers = TRUE,
                                          bounds = list(global = c(1, Inf))))    

findFreqTerms(opinions.tdm, lowfreq = 100, highfreq = Inf)

ft <- findFreqTerms(opinions.tdm, lowfreq = 100, highfreq = Inf)
as.matrix(opinions.tdm[ft,]) 

ft.tdm <- as.matrix(opinions.tdm[ft,])
sort(apply(ft.tdm, 1, sum), decreasing = TRUE)
a<-sort(apply(ft.tdm, 1, sum), decreasing = TRUE)
a<-as.data.frame(a)
a$word<-rownames(a)
aa<-a %>%  filter(a > 200) %>%
  mutate(word = reorder(word, a))

Answer 1

基本上，以下内容将带您完成 quanteda。您可能会添加一些行以进行额外的文本清理。 quanteda（和下面的 ranger）的优点是它们自动并行运行。使用 quanteda，您可以使用 quanteda_options(threads = x) 在 x 核上运行。默认为 2。

library(readtext)
library(quanteda)
library(stopwords)

files <- list.files(pattern = "pdf$")

# use option encoding = "UTF-8" if you encounter encoding issues
dat <- readtext(files)

my_dfm <- dat %>% 
  corpus() %>% 
  tokens(remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE) %>%
  tokens_tolower() %>%  
  tokens_remove(stopwords("tr", source = "stopwords-iso")) %>%
  tokens_wordstem(language = "turkish") %>% 
  dfm()

接下来，您可以使用 quanteda 中的 convert 函数将 dfm 转换为您需要的任何内容，作为 randomForest 的输入。阅读转换功能的帮助。请注意，quanteda 本身已经包含朴素贝叶斯。但是给你一个随机森林的例子，用 ranger 包来提高速度。

# transpose the document term matrix into a term document matrix. 
x <- t(convert(my_dfm, to = "matrix")) 
# transform into data.frame. 
my_df <- data.frame(features = rownames(x), x, stringsAsFactors = FALSE) 

library(ranger)
ranger(features ~ ., data = my_df)

Ranger result

Call:
 ranger(features ~ ., data = my_df) 

Type:                             Classification 
Number of trees:                  500 
Sample size:                      2547 
Number of independent variables:  6 
Mtry:                             2 
Target node size:                 1 
Variable importance mode:         none 
Splitrule:                        gini 
OOB prediction error:             100.00 %