将字符串拆分为较小的字符串以在数据框中创建新行（在 R 中）答案

【问题标题】：Split strings into smaller ones to create new rows in a data frame (in R)将字符串拆分为较小的字符串以在数据框中创建新行（在 R 中）
【发布时间】：2018-10-29 21:00:41
【问题描述】：

我是一个新的 R 用户，我目前正在努力解决如何在数据框的每一行中拆分字符串，然后使用修改后的字符串创建一个新行（以及修改原始字符串）。这是下面的示例，但实际数据集要大得多。

library(dplyr)
library(stringr)
library(tidyverse)
library(utils)

posts_sentences <- data.frame("element_id" = c(1, 1, 2, 2, 2), "sentence_id" = c(1, 2, 1, 2, 3), 
                "sentence" = c("You know, when I grew up, I grew up in a very religious family, I had the same sought of troubles people have, I was excelling in alot of ways, but because there was alot of trouble at home, we were always moving around", "Im at breaking point.I have no one to talk to about this and if I’m honest I think I’m too scared to tell anyone because if I do then it becomes real.I dont know what to do.", "I feel like I’m going to explode.", "I have so many thoughts and feelings inside and I don't know who to tell and I was going to tell my friend about it but I'm not sure.", "I keep saying omg!it's too much"), 
                "sentence_wc" = c(60, 30, 7, 20, 7), stringsAsFactors=FALSE)

我想分解超过特定字数的句子（此数据集为 15 个），使用正则表达式从较长的句子中创建新句子，以便首先尝试按句点（或其他符号）分解它)，然后如果字数仍然太长，我尝试逗号后跟一个 I（或大写字母），然后我尝试 'and' 后跟一个大写字母等。每次我创建一个新句子时，都需要将句子从旧行更改为句子的第一部分，同时更改字数（我有一个函数），同时创建一个具有相同元素 id 的新行，一个句子 id 位于序列后面（如果 sentence_id 为 1，则现在新句子为 2)，新句子字数，然后将以下所有句子更改为下一个 sentence_id 编号。

我已经为此工作了几天，但不知道该怎么做。我尝试过使用 unnest 令牌、str_split/extract 和各种 dplyr 过滤器、变异等组合以及 google/SO 搜索。有谁知道实现这一目标的最佳方法？ Dplyr 是首选，但我愿意接受任何可行的方法。如果您需要任何说明，请随时提出问题！

编辑以添加预期的输出数据框：

expected_output <- data.frame("element_id" = c(1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2), "sentence_id" = c(1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6), 
                                   "sentence" = c("You know, when I grew up", "I grew up in a very religious family", "I had the same sought of troubles people have", "I was excelling in alot of ways, but because there was alot of trouble at home, we were always moving around", "Im at breaking point.", "I have no one to talk to about this and if I’m honest I think I’m too scared to tell anyone because if I do then it becomes real.", "I dont know what to do.", "I feel like I’m going to explode.", "I have so many thoughts and feelings inside and", "I don't know who to tell and", "I was going to tell my friend about it but I'm not sure.", "I keep saying omg!", "it's too much"), 
                                   "sentence_wc" = c(6, 8, 8, 21, 4, 27, 6, 7, 9, 7, 13, 4, 3), stringsAsFactors=FALSE)

【问题讨论】：

请显示您的预期输出
@akrun 我刚做了，谢谢提醒！不知道该怎么做才能真正看到表格，就像我在其他问题中看到的那样。
您的expected_output 有两行sentence_wc > 15。为什么这些线没有被进一步分解？另外，你能提供你的实际正则表达式吗？ “句点（或其他符号）”太模糊了，特别是如果直到第二组才考虑逗号（“逗号后跟...”）。什么算作“其他符号”？
除了字数限制还有字数限制吗？
您的预期输出中仍有超过 15 个单词的句子？你有启发式方法来进一步分解它们吗？

标签： r string dplyr

【解决方案1】：

编辑：我已经编辑了整个答案以更详细地解决具体问题。

这并不完全是通用的，因为它假定组仅基于 element_id。

split_too_long <- function(str, max.words=15L, ...) {
  cuts <- stringi::stri_locate_all_words(str)[[1L]]

  # return one of these
  if (nrow(cuts) <= max.words) {
    c(str, NA_character_)
  }
  else {
    left <- substr(str, 1L, cuts[max.words, 2L])
    right <- substr(str, cuts[max.words + 1L, 1L], nchar(str))
    c(left, right)
  }
}

recursive_split <- function(not_done, done=NULL, ...) {
  left_right <- split_too_long(not_done, ...)

  # return one of these
  if (is.na(left_right[2L]))
    c(done, left_right[1L])
  else
    recursive_split(left_right[2L], done=c(done, left_right[1L]), ...)
}

collapse_split <- function(sentences, regex="[.;:] ?", ...) {
  sentences <- paste(sentences, collapse=". ")
  sentences <- unlist(strsplit(sentences, split=regex))
  # return
  unlist(lapply(sentences, recursive_split, done=NULL, ...))
}

group_fun <- function(grouped_df, ...) {
  # initialize new data frame with new number of rows
  new_df <- data.frame(sentence=collapse_split(grouped_df$sentence, ...),
                       stringsAsFactors=FALSE)
  # count words
  new_df$sentence_wc <- stringi::stri_count_words(new_df$sentence)
  # add sentence_id
  new_df$sentence_id <- 1L:nrow(new_df)
  # element_id must be equal because it is a grouping variable,
  # so take 1 to repeat it in output
  new_df$element_id <- grouped_df$element_id[1L]
  # return
  dplyr::filter(new_df, sentence_wc > 0L)
}

out <- posts_sentences %>%
  group_by(element_id) %>%
  do(group_fun(., max.words=5L, regex="[.;:!] ?"))

【讨论】：

感谢适用于所有句子的简单答案！但是，我希望只拆分单词数大于某个数字（例如 15）的句子，并让其余句子与以前相同。关于如何修改您的答案以使其工作的任何想法？
鉴于我目前的答案，我可以假设grouped_df$sentence 中的每个元素已经是一个完整的句子吗？所以我不必在决定它们是否太长之前将它们合并成一个长字符？
没关系，我已经更新了我的答案。您将需要 stringi 包。

【解决方案2】：

这是一种tidyverse 方法，可让您指定自己的启发式方法，我认为这应该最适合您的情况。关键是使用pmap 来创建每一行的列表，然后在必要时使用map_if 进行拆分。在我看来，这种情况仅靠dplyr 很难解决，因为我们在操作中添加了行，所以rowwise 很难使用。

split_too_long()的结构基本上是：

使用dplyr::mutate和tokenizers::count_words获取每个句子的字数
使用purrr::pmap 使每一行成为列表的元素，它接受数据框作为列列表作为输入
使用purrr::map_if检查字数是否大于我们想要的限制
如果满足上述条件，使用tidyr::separate_rows将句子分成多行，
然后用新的字数替换字数，并用filter（由双重分隔符创建）删除任何空行。

然后我们可以将其应用于不同的分隔符，因为我们意识到需要进一步拆分元素。在这里，我使用与您提到的启发式相对应的这些模式：

"[\\.\\?\\!] ?" 匹配任何 .!? 和可选空格
", ?(?=[:upper:])" 匹配 ,，可选空格，在大写字母之前
"and ?(?=[:upper:])" 匹配 and 可选空格，位于大写字母之前。

它正确地返回与您预期输出中相同的拆分句子。 sentence_id 很容易用row_number 重新添加到末尾，错误的前导/尾随空格可以用stringr::str_trim 删除。

注意事项：

我写这篇文章是为了在探索性分析中具有可读性，因此每次都拆分为列表并重新绑定在一起。如果您提前决定要使用哪些分隔符，可以将其放入一个 map 步骤中，这可能会使其更快，尽管我没有在大型数据集上对此进行分析。
根据 cmets，在这些拆分之后仍有超过 15 个单词的句子。您必须决定要拆分哪些其他符号/正则表达式以进一步缩短长度。
目前列名被硬编码为split_too_long。如果能够在函数调用中指定列名对您很重要，我建议您查看 programming with dplyr 小插图（只需进行一些调整即可实现）

posts_sentences <- data.frame(
  "element_id" = c(1, 1, 2, 2, 2), "sentence_id" = c(1, 2, 1, 2, 3),
  "sentence" = c("You know, when I grew up, I grew up in a very religious family, I had the same sought of troubles people have, I was excelling in alot of ways, but because there was alot of trouble at home, we were always moving around", "Im at breaking point.I have no one to talk to about this and if I’m honest I think I’m too scared to tell anyone because if I do then it becomes real.I dont know what to do.", "I feel like I’m going to explode.", "I have so many thoughts and feelings inside and I don't know who to tell and I was going to tell my friend about it but I'm not sure.", "I keep saying omg!it's too much"),
  "sentence_wc" = c(60, 30, 7, 20, 7), stringsAsFactors = FALSE
)

library(tidyverse)
library(tokenizers)
split_too_long <- function(df, regexp, max_length) {
  df %>%
    mutate(wc = count_words(sentence)) %>%
    pmap(function(...) tibble(...)) %>%
    map_if(
      .p = ~ .$wc > max_length,
      .f = ~ separate_rows(., sentence, sep = regexp)
      ) %>%
    bind_rows() %>%
    mutate(wc = count_words(sentence)) %>%
    filter(wc != 0)
}

posts_sentences %>%
  group_by(element_id) %>%
  summarise(sentence = str_c(sentence, collapse = ".")) %>%
  ungroup() %>%
  split_too_long("[\\.\\?\\!] ?", 15) %>%
  split_too_long(", ?(?=[:upper:])", 15) %>%
  split_too_long("and ?(?=[:upper:])", 15) %>%
  group_by(element_id) %>%
  mutate(
    sentence = str_trim(sentence),
    sentence_id = row_number()
  ) %>%
  select(element_id, sentence_id, sentence, wc)
#> # A tibble: 13 x 4
#> # Groups:   element_id [2]
#>    element_id sentence_id sentence                                      wc
#>         <dbl>       <int> <chr>                                      <int>
#>  1          1           1 You know, when I grew up                       6
#>  2          1           2 I grew up in a very religious family           8
#>  3          1           3 I had the same sought of troubles people ~     9
#>  4          1           4 I was excelling in alot of ways, but beca~    21
#>  5          1           5 Im at breaking point                           4
#>  6          1           6 I have no one to talk to about this and i~    29
#>  7          1           7 I dont know what to do                         6
#>  8          2           1 I feel like I’m going to explode               7
#>  9          2           2 I have so many thoughts and feelings insi~     8
#> 10          2           3 I don't know who to tell                       6
#> 11          2           4 I was going to tell my friend about it bu~    13
#> 12          2           5 I keep saying omg                              4
#> 13          2           6 it's too much                                  3

由reprex package (v0.2.0) 于 2018 年 5 月 21 日创建。

【讨论】：

感谢这个冗长而详细的回答，它非常有帮助。但是，当我使用上面的示例数据在我的计算机上运行它时，我得到了这个错误：Error in captureDots(strict = __quosured ) : the argument has already been evaluated 26. captureDots(strict = __quosured ) 25. dots_capture(..., __interp_lhs = __interp_lhs ) 24. dots_enquose(...) 23. quos(..., .named = TRUE) 22. tibble(...) at sample_dataset.R#35 21. .f(element_id = .l[[c(1L, i)]], sentence = .l[[c(2L, i)]], wc = .l[[c(3L, i)]], ...)你知道为什么会发生这个错误吗？
我不知道，会检查你所有的包版本以及你的 R 版本。您可能需要更新 tidyverse 包以及 rlang
作为参考，我有来自 github 的 R 3.5.0、tidyverse 1.2.1 和 rlang 0.2.0.9001，尽管我希望发布版本能够工作
感谢您的提示！我将 R 版本更新为 3.5.0（我认为这是问题所在）和软件包，现在它可以工作了。非常感谢您花时间帮助我解决这个问题！

【解决方案3】：

替代tidyverse解决方案：

library(dplyr)
library(tidyr)
library(stringr)
library(tidyverse)
library(utils)

check_and_split <- function(element_id, sentence_id, sentence, sentence_wc,
                             word_count, attmpt){

  methods <- c("\\.", ",\\s?(?=[I])", "and\\s?(?=[A-Z])")
  df <- data.frame(element_id=element_id,
             sentence_id=sentence_id,
             sentence=sentence,
             sentence_wc=sentence_wc,
             word_count=word_count,
             attmpt=attmpt,
             stringsAsFactors = FALSE)

    if(word_count<=15 | attmpt>=3){
      return(df) #early return
    } else{
     df %>% 
        tidyr::separate_rows(sentence, sep=methods[attmpt+1]) %>% 
        mutate(word_count=str_count(sentence,'\\w+'),
               attmpt = attmpt+1)
    }
}

posts_sentences %>% 
  mutate(word_count=str_count(sentence,'\\w+'),
         attmpt=0) %>%
  pmap_dfr(check_and_split) %>% 
  pmap_dfr(check_and_split) %>% 
  pmap_dfr(check_and_split)

在这里，我们创建了一个辅助函数，它接收一行（由元素分解，由 purrr::pmap() 提供服务），我们将其组装回数据框中，检查字数是否超过 15 以及尝试尝试的次数前句。然后我们使用tidyr::separate_rows()和下一次尝试对应的分隔令牌，更新word_count和number of attempts并返回数据帧。

我正在应用相同的函数 3 次 - 这可能会被包装成一个循环（lapply/purrr::map 将不起作用，因为我们需要按顺序更新数据帧）。

就正则表达式标记而言，首先我们使用文字.，然后我们跟踪逗号和零个或多个空格，然后是“I”。请注意积极的前瞻语法。最后，我们尝试使用“and”，可能是空格，前瞻后跟大写字母。

希望这是有道理的

【讨论】：

【解决方案4】：

此解决方案首先在大写字母前用逗号或句点分隔句子。然后用逗号和句号分割句子。最后，如果一个句子仍然高于限制词。句子由每个大写字母分隔。

posts_sentences <- data.frame("element_id" = c(1, 1, 2, 2, 2), "sentence_id" = c(1, 2, 1, 2, 3), 
                              "sentence" = c("You know, when I grew up, I grew up in a very religious family, I had the same sought of troubles people have, I was excelling in alot of ways, but because there was alot of trouble at home, we were always moving around", "Im at breaking point.I have no one to talk to about this and if I’m honest I think I’m too scared to tell anyone because if I do then it becomes real.I dont know what to do.", "I feel like I’m going to explode.", "I have so many thoughts and feelings inside and I don't know who to tell and I was going to tell my friend about it but I'm not sure.", "I keep saying omg!it's too much"), 
                              "sentence_wc" = c(60, 30, 7, 20, 7), stringsAsFactors=FALSE)

# To create an empty data frame to save the new elements

new_posts_sentences <- data.frame(element_id = as.numeric(),
                 sentence_id =as.numeric(), 
                 sentence = character(), 
                 sentence_wc = as.numeric(),  stringsAsFactors=FALSE) 

limit_words <- 15 # 15 for this data set

countSentences <- 0

for (sentence in posts_sentences[,3]) {

        vector <- character()

        Velement_id <- posts_sentences$element_id[countSentences + 1]

        vector <- c(vector, sentence) #To create a vector with the sentences
        vector <- vector[!vector %in% ''] #remove empty elements from vector

        ## First we will separate the sentences that start with a uppercase after of a capital letter
        if(lengths(gregexpr("[A-z]\\W+", sentence)) > limit_words ){

                vector <- vector[!vector %in% sentence]

                split_points <- unlist(gregexpr("[:,:]\\s[A-Z]", sentence)) # To get the character position

                ## If a sentences is still over the limit words value. Let's split it for each comma or period
                sentences_1 <- substring(sentence, c(1, split_points + 2), c(split_points -1, nchar(sentence)))

                for(sentence in sentences_1){

                        vector <- c(vector, sentence)
                        vector <- vector[!vector %in% '']

                        if(lengths(gregexpr("[A-z]\\W+", sentence)) > limit_words){

                                vector <- vector[!vector %in% sentence]

                                split_points <- unlist(gregexpr("[:,:]|[:.:]", sentence))

                                sentences_2 <- substring(sentence, c(1, split_points + 1), c(split_points -1, nchar(sentence)))

                                ## If a sentence is still s still over the limit words value. Let's split it for each capital letter

                                for(sentence in sentences_2){

                                        vector <- c(vector, sentence)
                                        vector <- vector[!vector %in% '']

                                        if(lengths(gregexpr("[A-z]\\W+", sentence)) > limit_words){

                                                vector <- vector[!vector %in% sentence]

                                                split_points <- unlist(gregexpr("[A-Z]", sentence))

                                                sentences_3 <- substring(sentence,c(1, split_points), c(split_points -1, nchar(sentence)))

                                                vector <- c(vector, sentences_3)
                                                vector <- vector[!vector %in% '']

                                        }

                                }

                        }

                }

        }

        ## To make a data frame o each original sentence
        element_id <- rep(Velement_id, length(vector))
        sentence_id <- 1:length(vector)
        sentence_wc <- character()
        for (element in vector){sentence_wc <- c(sentence_wc, (lengths(gregexpr("[A-z]\\W+", element)))) }
        sentenceDataFrame <- data.frame(element_id, sentence_id, vector, sentence_wc)       

        ## To join it with the final dataframe
        new_posts_sentences <- rbind(new_posts_sentences, sentenceDataFrame)

        countSentences <- countSentences + 1

}

你得到这个数据框

print(new_posts_sentences)

   element_id sentence_id                                           vector sentence_wc
1           1           1                         You know, when I grew up           5
2           1           2             I grew up in a very religious family           7
3           1           3    I had the same sought of troubles people have           8
4           1           4                  I was excelling in alot of ways           6
5           1           5    but because there was alot of trouble at home           8
6           1           6                     we were always moving around           4
7           1           1                             Im at breaking point           3
8           1           2      I have no one to talk to about this and if           11
9           1           3                                      I’m honest            3
10          1           4                                         I think            2
11          1           5        I’m too scared to tell anyone because if            9
12          1           6                        I do then it becomes real           5
13          1           7                           I dont know what to do           5
14          2           1                I feel like I’m going to explode.           8
15          2           1 I have so many thoughts and feelings inside and            9
16          2           2                    I don't know who to tell and            8
17          2           3      I was going to tell my friend about it but           10
18          2           4                                     I'm not sure           3
19          2           1                  I keep saying omg!it's too much           7

希望对你有帮助。

【讨论】：

【解决方案5】：

我认为最简单的方法是使用 stringr 包中的 str_split() 函数（根据您的正则表达式拆分每个文本块），然后使用 tidyr 包中的 unnest() 函数。

sentences_split = posts_sentences %>%
  mutate(text_split=str_split(sentence, pattern = "\\.")) %>%
  unnest(text_split) %>%

  #Count number of words in text_split
  mutate(wc_split = str_count(text_split, "\\w+")) %>%

  filter(wc_split!=0) %>%

  #Split again if text_split column has >15 words
  mutate(text_split_again = ifelse(wc_split>15, str_split(text_split, pattern = ",\\sI"), text_split)) %>%
  unnest(text_split_again)

【讨论】：