如何在R中全文字符串模式之间提取唯一字符串？答案

【问题标题】：How to extract unique string in between string pattern in full text in R?如何在R中全文字符串模式之间提取唯一字符串？
【发布时间】：2020-05-08 03:40:51
【问题描述】：

我希望从以下文本中提取在国会面前作证的人的姓名和职业：

text

全文见这里：https://www.congress.gov/116/crpt/srpt19/CRPT-116srpt19.pdf

似乎这些名字介于“听到证词”和下一个“.”之间。那么，如何提取这两种模式之间的名称呢？文本要长得多（50 页文档），但我想如果我能写一个，我会为其余的文本做。

我知道我不能使用 NLP 来提取姓名，因为它们是例如没有作证的人的姓名。

【问题讨论】：

您在问题中输入的数据非常好，但我们很难将其放入 R 会话中。您能否运行dput(yourdata) 并将结果粘贴到您的问题中，以便其他人可以轻松使用数据
另见stackoverflow.com/questions/39086400/…

标签： r regex text nlp

【解决方案1】：

NLP 可能是不可避免的，因为文本中有许多缩写。试试这个工作流程：

按句子分词
删除没有“见证”的句子
从剩余句子中提取人物 + 职业

有几个带有句子标记器的软件包，但是 openNLP 在处理包含缩写的句子时通常对我来说效果最好。下面的代码应该能让你接近你的目标：

library(tidyverse)
library(pdftools)
library(openNLP)

# Get the data
testimony_url <- "https://www.congress.gov/116/crpt/srpt19/CRPT-116srpt19.pdf"
download.file(testimony_url, "testimony.pdf")
text_raw <- pdf_text("testimony.pdf")

# Clean the character vector and smoosh into one long string.
text_string <- str_squish(text_raw) %>% 
    str_replace_all("- ", "") %>% 
    paste(collapse = " ") %>% 
    NLP::as.String()

# Annotate and extract the sentences.
annotations <- NLP::annotate(text_string, Maxent_Sent_Token_Annotator())
sentences <- text_string[annotations]

# Some sentences starting with "Testimony" list multiple persons. We need to
# split these and clean up a little.
name_title_vec <- str_subset(sentences, "Testimony was") %>% 
    str_split(";") %>% 
    unlist %>% 
    str_trim %>% 
    str_remove("^(Testimony .*? from|and) ") %>% 
    str_subset("^\\(\\d\\)", negate = T)

# Put in data frame and separate name from profession/title.
testimony_tibb <- tibble(name_title_vec) %>% 
    separate(name_title_vec, c("name", "title"), sep = ", ", extra = "merge")

您应该最终得到以下数据框。可能需要进行一些额外的清洁：

# A tibble: 95 x 2
   name                       title                                                               
   <chr>                      <chr>                                                               
 1 the Honorable John Koskin… Commissioner, Internal Revenue Service, Washington, DC.             
 2 the Honorable Steven Mnuc… Secretary of the Treasury, United States Department of the Treasury…
 3 the Honorable Jonathan Ta… former Assistant Secretary for Tax Policy 2000–2001, United States …
 4 the Honorable Pamela F. O… former Assistant Secretary for Tax Policy 2002–2004, United States …
 5 the Honorable Eric Solomon former Assistant Secretary for Tax Policy 2006–2009, United States …
 6 the Honorable Mark J. Maz… "former Assistant Secretary for Tax Policy 2012–2017, United States…
 7 Mr. Daniel Garcia-Diaz     Director, Financial Markets and Community Investment, United States…
 8 Mr. Grant S. Whitaker      president, National Council of State Housing Agencies, Washington, …
 9 the Honorable Katherine M… Ph.D., professor of public policy and planning, and faculty directo…
10 Mr. Kirk McClure           Ph.D., professor, Urban Planning Program, School of Public Policy a…
# … with 85 more rows

【讨论】：