【问题标题】:Abstract multiple keywords from text and print in data frame从文本中提取多个关键字并在数据框中打印
【发布时间】:2019-03-01 04:18:03
【问题描述】:

我有一个数据框(称为 all_data),如下所示:

Title         Text 
Title_1       Very interesting word_1 and also keyword_2
Title_2       hello keyword_1, and keyword_3. 

我还有第二个数据框(称为关键字),如下所示:

keywords
word_1
word_2
word_3
word_4a word_4b word_4c

我想在 all_data 数据框中创建一个额外的列。在此列中,如果关键字之一(来自关键字数据框)出现在 all_data$Text 或 all_data$Title 列中,我想打印相关关键字。例如:

Title         Text                                               Keywords
Title_1       Very interesting word_1 and also word_2, word_1.   word_1, word_2
Title_2       hello word_1, and word_3.                          word_1, word_3
Title_3       difficult! word_4b, and word_4a also word_4c       word_4a word_4b word_4c

!只需在 all_data$Words 列中打印一次单词,而不是多次。 对我来说,更难的部分是打印一个“关键字”,例如:“keyword_A Keyword_A1 Keyword_A3”,只有当关键字的所有部分都出现在相关文本中时才会出现。

这里回答了这个问题(Recognize patterns in column, and add them to column in Data frame),我在这里使用了DJack他的解决方案:

ls <- strsplit(tolower(paste(all_data$Title, all_data$Text)),"(\\s+)|(?!')(?=[[:punct:]])", perl = TRUE)    

all_data$Keywords <- do.call("rbind",lapply(ls,function(x) paste(unique(x[x %in% tolower(keywords)]), collapse = ", ")))

但是当多个关键字出现时它会失败(一个关键字,比如:老奶奶,如果你有这样的文字应该出现:“嘿,你的奶奶很好,而且很老”。

更新

@Nicolas2 帮我解决了问题(谢谢)。但不幸的是它失败了。任何人都知道如何解决这个问题?正如您在下面的示例中所看到的,例如,关键字“feyenoord skin”不应出现(因为“skin”没有出现在文本中)。我只希望关键字出现在文本中(或者有多个关键字,例如“Hello World”,如果所有单词都出现在文本中,那就太好了(所以 Hello and World)。非常感谢!

df <- data.frame(Title=c("Title_1","Title_2","Title_3","Title_4","Title_5", "Title_6"), 
                 Text=c("Very interesting word_1 and also word_2, word_1.", 
                        "hello word_1, and word_3.", 
                        "difficult! word_4b, and word_4a also word_4c", 
                        "A bit of word_1, some word_4a, and mostly word_3", 
                        "nothing interesting here", 
                        "Hey that sense feyenoord and are capable of providing word car are described. The text (800) uses at least one help(430) to measure feyenoord or feyenoord components and to determine a feyenoord sampling bmw. The word car is rstudio, at least in part, using the feyenoord sampling bmw. The feyenoord sampling bmw may be rstudio, at least in part, using a feyenoord volume (640) and/or a feyenoord generation bmw, both of which may be python or prerstudio."), 
                 stringsAsFactors=F) 


keywords<-data.frame(Keyword=c("word_1","word_2","word_3","word_4a word_4b word_4c", 
                               "a feyenoord sense", 
                               "feyenoord", "feyenoord feyenoord", "feyenoord skin", "feyenoord collection", 
                               "skin feyenoord", "feyenoord collector", "feyenoord bmw", 
                               "collection feyenoord", "concentration feyenoord", "feyenoord sample",
                               "feyenoord stimulation", "analyte feyenoord", "collect feyenoord", 
                               "feyenoord collect", "pathway feyenoord feyenoord sandboxs", 
                               "feyenoord bmw mouses", "sandbox", "bmw", 
                               "pulse bmw three levels"),stringsAsFactors=F) 

# split the keywords into words, but remember keyword length 
k <- keywords %>% mutate(l=str_split(Keyword," ")) %>% unnest %>% 
  group_by(Keyword) %>% mutate(n=n()) %>% ungroup 
# split the title into words 
# compare with words from keywords 
# keep only possibly multiple, but full matches 
# collate all results and merge back to the original data 
test <- df %>% mutate(l=str_split(Text,"[ .,]")) %>% unnest %>% 
  inner_join(k,by="l") %>% 
  group_by(Title,Keyword) %>% filter(n()%%n==0) %>% 
  distinct(Keyword) %>% ungroup %>% nest(Keyword) %>% 
  rowwise %>% mutate(keywords=paste(data[[1]],collapse=", ")) %>% select(-data) %>% 
  inner_join(df,.,by="Title") 

View(test)

【问题讨论】:

  • 我很确定我以前回答过一个与此完全相同的问题。更努力地尝试搜索功能!
  • 无法在您个人资料的“答案”中找到它。在常规搜索栏中也找不到。

标签: r regex text dplyr


【解决方案1】:
Title <- c("A","B","C","A","A","B","A","A","B","C")
Text <- c("A",11,12,13,14,15,14,13,12,"hi")
df <- data.frame(Title,Text, stringsAsFactors=FALSE)

keywords <- c("A","B","hi")
keys <- data.frame(keywords,stringsAsFactors=FALSE)

这是一个有点长的逻辑链,很难阅读。但它确实是一种变异,干净且快速。

require(dplyr)
require(stringr)
df %>% mutate(Keywords = paste(str_c(keys$keywords[which(keys$keywords %in% 
df$Title)],collapse = ","),str_c(keys$keywords[which(!keywords %in% 
df$Title)] 
[which(keywords[which(!keywords %in% df$Title)] %in% df$Text)], 
collapse=","), 
sep=",")) -> df

让我分解一下,在粘贴中我们有两个术语,第一个是

str_c(keys$keywords[which(keys$keywords %in% df$Title)],collapse = ",")

$Title 列中找到关键字并需要str_c 将找到的关键字连接到一个字符串中,以避免由于未连接的结果是数据帧而不是字符串而造成的混乱重复。下一个词是:

str_c(keys$keywords[which(!keywords %in% df$Title)][which(keywords[which(!keywords 
%in% df$Title)] %in% df$Words)], collapse=",")

这看起来很糟糕,但正在调用不在$Title 中的关键字,而这些关键字在$Text 中。这段相当长的逻辑是必要的,这样我们就不会重复我们在$Title 中看到的关键字。出于同样的原因,我们应该使用str_c 来进行字符串输出。然后两个字符串的粘贴给我们你想要的输出。修改collapse=" ,"sep = " ," 可以根据需要添加空格。

【讨论】:

    【解决方案2】:

    我没有费心优化任何东西,只是做了最直接的事情:

    library(data.table)
    
    setDT(df)
    setDT(keywords)
    
    keywords[, strsplit(Keyword, ' '), by = Keyword
           ][, c(.SD[, .(row = seq_len(nrow(df)), found = grepl(V1, df$Text)), by = V1],
                 N = .N), by = Keyword
           ][, sum(found) == N[1], by = .(Keyword, row)
           ][, paste(Keyword[V1], collapse = ","), by = row]
    #   row                                            V1
    #1:   1                                 word_1,word_2
    #2:   2                                 word_1,word_3
    #3:   3                       word_4a word_4b word_4c
    #4:   4                                 word_1,word_3
    #5:   5                                              
    #6:   6 a feyenoord sense,feyenoord,feyenoord bmw,bmw
    

    【讨论】:

      【解决方案3】:

      如果关键字仅由一个单词组成,例如“old grandma”可以由“old”和“grandma”两个关键字组成,那么使用非常适合文本分析的包的解决方案呢?例如tidytext

      library(dplyr)     
      library(tidytext)  # text manipulation
      

      首先,我们必须将我们的数据作为每个单词都是一行,所以我们以这种方式拆分 all_data 和关键字:

      all_data_un <- all_data %>% unnest_tokens(word,Text)
          > all_data_un
             Title        word
      1    Title_1        very
      1.1  Title_1 interesting
      1.2  Title_1      word_1
      1.3  Title_1         and
      1.4  Title_1        also
      1.5  Title_1      word_2
      1.6  Title_1      word_1
      2    Title_2       hello
      2.1  Title_2      word_1
      2.2  Title_2         and
      2.3  Title_2      word_3
      3    Title_3   difficult
      3.1  Title_3     word_4b
      3.2  Title_3         and
      3.3  Title_3     word_4a
      3.4  Title_3        also
      ....
      
      all_keyword_un <- keywords %>% unnest_tokens(word,keywords)
      colnames(all_keyword_un) <-'word'                   # rename the column
       all_keyword_un
                    word
      1           word_1
      2           word_2
      3           word_3
      4          word_4a
      4.1        word_4b
      4.2        word_4c
      5                a
      5.1      feyenoord
      5.2          sense
      6        feyenoord
      7        feyenoord
      7.1      feyenoord
      8        feyenoord
      8.1           skin
      9        feyenoord
      9.1     collection
      10            skin
      10.1     feyenoord
      11       feyenoord
      11.1     collector
      12       feyenoord
      12.1           bmw
      13      collection
      13.1     feyenoord
      ....
      

      如您所见,unnest_tokens() 会在必要时删除标点符号和大写字母。

      现在可以只过滤关键字中的单词:

      all_data_un_fi <- all_data_un[all_data_un$word %in% all_keyword_un$word,]
            > all_data_un_fi
             Title      word
      1.2  Title_1    word_1
      1.5  Title_1    word_2
      1.6  Title_1    word_1
      2.1  Title_2    word_1
      2.3  Title_2    word_3
      3.1  Title_3   word_4b
      3.3  Title_3   word_4a
      3.5  Title_3   word_4c
      4    Title_4         a
      4.3  Title_4    word_1
      4.5  Title_4   word_4a
      4.8  Title_4    word_3
      6.2  Title_6     sense 
      ....
      

      最后一步:合并数据集和每个句子中找到的关键字:

      all_data %>%                                      # starting data
      left_join(all_data_un_fi) %>%                     # joining without forget any sentence
      group_by(Title,Text) %>%                          # group by title and text
      summarise(keywords = paste(word, collapse =','))  # put in one cell all the keywords finded
      
      
         Joining, by = "Title"
      # A tibble: 6 x 3
      # Groups:   Title [?]
        Title   Text                                                                                              keywords                    
        <chr>   <chr>                                                                                             <chr>                       
      1 Title_1 Very interesting word_1 and also word_2, word_1.                                                  word_1,word_2,word_1        
      2 Title_2 hello word_1, and word_3.                                                                         word_1,word_3               
      3 Title_3 difficult! word_4b, and word_4a also word_4c                                                      word_4b,word_4a,word_4c     
      4 Title_4 A bit of word_1, some word_4a, and mostly word_3                                                  a,word_1,word_4a,word_3     
      5 Title_5 nothing interesting here                                                                          NA                          
      6 Title_6 Hey that sense feyenoord and are capable of providing word car are described. The text (800) use~ sense,feyenoord,feyenoord,f~
      

      用一个或多个单词组成的关键字,所以“老奶奶”的关键字是“老奶奶”,你可以这样做:

      library(stringr)
      library(dplyr)
      

      首先是一个空列表:

      mylist <- list()
      

      然后你可以用循环填充它,对于每个关键字,找到包含该关键字的句子:

      for (i in keywords$keywords) {
      keyworded <- all_data %>%filter(str_detect(Text, i)) %>% mutate(keyword = i)
        mylist[[i]] <- keyworded}
      

      把它放在一个data.frame中:

       df <- do.call("rbind",mylist)%>%data.frame()
      

      然后按每个关键字分组:

       df %>% group_by(Title,Text) %>% summarise(keywords = paste(keyword,collapse=','))
      
      # A tibble: 4 x 3
      # Groups:   Title [?]
        Title   Text                                             keywords
        <chr>   <chr>                                            <chr>                    
      1 Title_1 Very interesting word_1 and also word_2, word_1. word_1,word_2            
      2 Title_2 hello word_1, and word_3.                        word_1,word_3            
      3 Title_4 A bit of word_1, some word_4a, and mostly word_3 word_1,word_3            
      4 Title_6 Hey that sense feyenoord and are capable of pro~ feyenoord,bmw,sense feye~
      

      注意:重复的内容像第一句一样被删除,word_4a 不在其中,因为在关键字中,您只能将它与其他单词放在一个字符串中。


      有数据(注意我修改了key添加“sense feyenoord”来测试keywords末尾两个词的关键字):

         all_data <-  data.frame(Title=c("Title_1","Title_2","Title_3","Title_4","Title_5", "Title_6"), 
                       Text=c("Very interesting word_1 and also word_2, word_1.", 
                              "hello word_1, and word_3.", 
                              "difficult! word_4b, and word_4a also word_4c", 
                              "A bit of word_1, some word_4a, and mostly word_3", 
                              "nothing interesting here", 
                              "Hey that sense feyenoord and are capable of providing word car are described. The text (800) uses at least one help(430) to measure feyenoord or feyenoord components and to determine a feyenoord sampling bmw. The word car is rstudio, at least in part, using the feyenoord sampling bmw. The feyenoord sampling bmw may be rstudio, at least in part, using a feyenoord volume (640) and/or a feyenoord generation bmw, both of which may be python or prerstudio."), 
                       stringsAsFactors=F) 
      
      keywords<-data.frame(keywords = c("word_1","word_2","word_3","word_4a word_4b word_4c", 
                                     "a feyenoord sense", 
                                     "feyenoord", "feyenoord feyenoord", "feyenoord skin", "feyenoord collection", 
                                     "skin feyenoord", "feyenoord collector", "feyenoord bmw", 
                                     "collection feyenoord", "concentration feyenoord", "feyenoord sample",
                                     "feyenoord stimulation", "analyte feyenoord", "collect feyenoord", 
                                     "feyenoord collect", "pathway feyenoord feyenoord sandboxs", 
                                     "feyenoord bmw mouses", "sandbox", "bmw", 
                                     "pulse bmw three levels","sense feyenoord"), stringsAsFactors=F)
      

      您也可以混合使用两种方式,同时获得两种结果,然后折叠在一起或创建它们的组合。


      编辑:
      要将它们合并在一起,您有很多方法,一个简单的方法是这样,它也可以输出唯一性:

      # first we create all the "single" keywords, i e "old grandma" -> "old" and "grandma"
      all_keyword_un <- keywords %>% unnest_tokens(word,keywords)
      colnames(all_keyword_un) <-'keywords'                   # rename the column
      
      # then you bind them to the full keywords, i.e. "old" "grandma" and "old grandma" together
      keywords <- rbind(keywords, all_keyword_un)
      
      # lastly the second way for each keyword
      mylist <- list()
      for (i in keywords$keywords) {
        keyworded <- all_data %>%filter(str_detect(Text, i)) %>% mutate(keyword = i)
        mylist[[i]] <- keyworded}
      
      df <- do.call("rbind",mylist)%>%data.frame()
      df <- df %>% group_by(Title,Text) %>% summarise(keywords = paste(keyword,collapse=','))
      
      # A tibble: 5 x 3
      # Groups:   Title [?]
        Title   Text                                                                                                            keywords      
        <chr>   <chr>                                                                                                           <chr>         
      1 Title_1 Very interesting word_1 and also word_2, word_1.                                                                word_1,word_2~
      2 Title_2 hello word_1, and word_3.                                                                                       word_1,word_3~
      3 Title_3 difficult! word_4b, and word_4a also word_4c                                                                    word_4a,word_~
      4 Title_4 A bit of word_1, some word_4a, and mostly word_3                                                                word_1,word_3~
      5 Title_6 Hey that sense feyenoord and are capable of providing word car are described. The text (800) uses at least one~ feyenoord,bmw~
      

      【讨论】:

      • 感谢 s_t!我多次尝试让您的解决方案在示例数据集上运行,但我几乎在每一行都面临错误。那是因为我检查错了吗?它在你的电脑上工作吗?再次感谢!!
      • 不客气!在第一个解决方案中,colnames(all_keyword_un) &lt;-'word' 行中有一个错字:现在一切正常,如果您有任何错误,请告诉我在哪里和哪个错误。我还更新了所有内容,只使用“feyenoord”数据。
      • 这太棒了!两种方法都有效。非常感谢,很抱歉问这个问题,但是是否也可以举一个例子来说明如何将它们折叠在一起?还是制作两列(带有关键字)并将它们合并?然后让一切变得独一无二?
      • 已编辑,添加了两种方法的混合有“老奶奶”、“老”和“奶奶”。
      【解决方案4】:
      df <- data.frame(
         Title=c("Title_1","Title_2","Title_3","Title_4"),
         Text=c("Very interesting word_1 and also word_2, word_1.",
                "hello word_1, and word_3.",                     
                "difficult! word_4b, and word_4a also word_4c",
                "nothing interesting here"),stringsAsFactors=FALSE)
      
      keywords<-data.frame(Keyword=c("word_1","word_2","word_3","word_4a word_4b word_4c"),stringsAsFactors=F)
      
      df %>% mutate(l=str_split(Text,"[ .,]")) %>% unnest %>%
        inner_join(keywords %>% mutate(l=str_split(Keyword," ")) %>% unnest, by="l") %>%
        select(-Keyword) %>% distinct %>% nest(l)
      #    Title                                             Text                      data
      #1 Title_1 Very interesting word_1 and also word_2, word_1.            word_1, word_2
      #2 Title_2                        hello word_1, and word_3.            word_1, word_3
      #3 Title_3     difficult! word_4b, and word_4a also word_4c word_4b, word_4a, word_4c
      

      因此,结果存储在列表中。将其转换为字符串:

      df %>% mutate(l=str_split(Text,"[ .,]")) %>% unnest %>%
        inner_join(keywords %>% mutate(l=str_split(Keyword," ")) %>% unnest,by="l") %>%
        select(-Keyword) %>% distinct %>% arrange(l) %>% nest(l) %>%
        rowwise %>% mutate(keywords=paste(data[[1]],collapse=" ")) %>% select(-data)
      ## A tibble: 3 x 3
      #  Title   Text                                             keywords               
      #  <chr>   <chr>                                            <chr>                  
      #1 Title_1 Very interesting word_1 and also word_2, word_1. word_1 word_2          
      #2 Title_2 hello word_1, and word_3.                        word_1 word_3          
      #3 Title_3 difficult! word_4b, and word_4a also word_4c     word_4a word_4b word_4c
      

      当关键字是多个单词时,删除部分匹配并将它们视为单个实体的升级版本:

      df <- data.frame(Title=c("Title_1","Title_2","Title_3","Title_4","Title_5"),
      Text=c("Very interesting word_1 and also word_2, word_1.",
             "hello word_1, and word_3.",                     
             "difficult! word_4b, and word_4a also word_4c",
             "A bit of word_1, some word_4a, and mostly word_3",
             "nothing interesting here"),
        stringsAsFactors=F)
        keywords<-data.frame(Keyword=c("word_1","word_2","word_3","word_4a word_4b word_4c"),stringsAsFactors=F)
      
      # split the keywords into words, but remember keyword length
      k <- keywords %>% mutate(l=str_split(Keyword," ")) %>% unnest %>%
         group_by(Keyword) %>% mutate(n=n()) %>% ungroup
      # split the title into words
      # compare with words from keywords
      # keep only possibly multiple, but full matches
      # collate all results and merge back to the original data
      df %>% mutate(l=str_split(Text,"[ .,]")) %>% unnest %>%
         inner_join(k,by="l") %>%
         group_by(Title,Keyword) %>% filter(n()%%n==0) %>%
         distinct(Keyword) %>% ungroup %>% nest(Keyword) %>%
         rowwise %>% mutate(keywords=paste(data[[1]],collapse=", ")) %>% select(-data) %>%
         inner_join(df,.,by="Title")
      #    Title                                             Text                keywords
      #1 Title_1 Very interesting word_1 and also word_2, word_1.          word_1, word_2
      #2 Title_2                        hello word_1, and word_3.          word_1, word_3
      #3 Title_3     difficult! word_4b, and word_4a also word_4c word_4a word_4b word_4c
      #4 Title_4    A bit word_1, some word_4a, and mostly word_3          word_1, word_3
      

      【讨论】:

      • 太棒了。但是如果你把你的代码分配给一个数据框(像这样:foo %.....)然后View(foo),你会看到列数据不正确(表示为:list( l = c("word_1", "word_2")) )。此外,我看到 word_4b、word_4a 和 word_4c 被表示为单独的单词。是否可以像这样打印:word_4a word4_b word 4_c(作为一个关键字)?再次:谢谢!
      • 像在关键字数据框中一样打印原始单词 :-) 所以:“word_4a word_4b word_4c”
      • @R 溢出我升级了我的答案。您是否认为分隔 word_1 和 word_2 的逗号相关?
      • 这太棒了!是的,现在答案可能是这样的:word_1 word_4a word_4b word_4c。我真正寻找的几乎是那个,但用逗号分隔(如:word_1,word_4a word_4b word_4c)。如您所见:关键字 word_4a word_4b word_4c 没有用逗号分隔。真的很感激,Nicolas2!
      • @R overflow 没有什么是不可能的,它只会使代码复杂化。您对“A bit word_1, some word_4a, and most word_3”这样的句子有什么期待?
      猜你喜欢
      • 1970-01-01
      • 2015-01-24
      • 2019-11-18
      • 2021-01-21
      • 1970-01-01
      • 2020-09-24
      • 1970-01-01
      • 2011-12-21
      • 2022-01-09
      相关资源
      最近更新 更多