我正在尝试使用 R 的 tm 包对我的文本数据进行矢量化。
目前我的数据语料是以下形式：
1. The sports team practiced today
2. The soccer team went took the day off
然后数据将被矢量化为：
<the, sports, team, practiced, today, soccer, went, took, off>
1. <1, 1, 1, 1, 1, 0, 0, 0, 0>
2. <1, 0, 1, 0, 0, 1, 1, 1, 1>
我更愿意为我的向量使用一组自定义短语，例如：
<sports team, soccer team, practiced today, day off>
1. <1, 0, 1, 0>
2. <0, 1, 0, 1>
R 中是否有可以执行此操作的包或函数？或者是否有任何其他具有类似功能的开源资源？谢谢。
【问题讨论】：
标签： r corpus text-analysis
您询问了其他文本包，我欢迎您尝试quanteda，这是我与 Paul Nulty 开发的。
在下面的代码中，首先将所需的多词短语定义为命名列表，使用dictionary() 构造函数将其键入为quanteda“字典”类对象，然后使用phrasetotoken() 进行转换在您的文本中找到的短语变成由下划线连接的短语单词组成的单个“标记”。标记器忽略下划线，因此您的短语会被视为单字标记。
dfm() 是文档特征矩阵的构造函数，可以采用定义要保留的特征的正则表达式，这里任何包含下划线字符的短语（正则表达式当然可以改进，但我故意保持简单这里）。 dfm() 有很多选择 -- 请参阅 ?dfm。
install.packages("quanteda")
library(quanteda)
mytext <- c("The sports team practiced today",
"The soccer team went took the day off")
myphrases <- dictionary(list(myphrases=c("sports team", "soccer team", "practiced today", "day off")))
mytext2 <- phrasetotoken(mytext, myphrases)
mytext2
## [1] "The sports_team practiced_today" "The soccer_team went took the day_off"
# keptFeatures is a regular expression: keep only phrases
mydfm <- dfm(mytext2, keptFeatures = "_", verbose=FALSE)
mydfm
## Document-feature matrix of: 2 documents, 4 features.
## 2 x 4 sparse Matrix of class "dfmSparse"
## features
## docs day_off practiced_today soccer_team sports_team
## text1 0 1 0 1
## text2 1 0 1 0
很乐意帮助解决任何与 quanteda 相关的问题，包括功能请求（如果您可以提出改进短语处理的建议）。
【讨论】：
这样的事情怎么样？
library(tm)
text <- c("The sports team practiced today", "The soccer team went took the day off")
corpus <- Corpus(VectorSource(text))
tokenizing.phrases <- c("sports team", "soccer team", "practiced today", "day off")
phraseTokenizer <- function(x) {
require(stringr)
x <- as.character(x) # extract the plain text from the tm TextDocument object
x <- str_trim(x)
if (is.na(x)) return("")
phrase.hits <- str_detect(x, ignore.case(tokenizing.phrases))
if (any(phrase.hits)) {
# only split once on the first hit, so we don't have to worry about multiple occurences of the same phrase
split.phrase <- tokenizing.phrases[which(phrase.hits)[1]]
# warning(paste("split phrase:", split.phrase))
temp <- unlist(str_split(x, ignore.case(split.phrase), 2))
out <- c(phraseTokenizer(temp[1]), split.phrase, phraseTokenizer(temp[2])) # this is recursive, since f() calls itself
} else {
out <- MC_tokenizer(x)
}
# get rid of any extraneous empty strings, which can happen if a phrase occurs just before a punctuation
out[out != ""]
}
tdm <- TermDocumentMatrix(corpus, control = list(tokenize = phraseTokenizer))
> Terms(tdm)
[1] "day off" "practiced today" "soccer team" "sports team" "the" "took"
[7] "went"
【讨论】：