R在使用rvest抓取表时将<p>元素的XML节点集转换为数据框

Question

我们正在从https://nbpa.com/agents/directory 提取联系信息。页面上没有表格，而是<div>s，里面有<p>元素：

我们可以通过以下方式获取这个元素：

agents_url <- "https://nbpa.com/agents/directory"
agents_page <- agents_url %>% read_html()
agents_page_elements <- agents_page %>% html_nodes('div.accordion-inner')

agents_page_elements[1]
agents_page_elements[1] %>% html_nodes('p')

我们希望将其转换为 1 行数据框：

Cell             Email                  Professional Credentials:
(123) 456-7890   firstlast@email.com    "NBA Certified Player Agent..."

这可能吗？这个网页抓取的挑战在于网站上的每个 accordion-inner div 都有不同的 p 元素。有些有 Cell: 和 Email:，其他有 Education:、Address: 等。它因手风琴内部而异。如果我们可以将每个单独的节点变成一个 1 行数据帧，我们就可以使用 plyr::rbind.fill() 将所有数据帧 rbind 在一起。

Answer 1

得到text后我们可以使用read.dcf

new <- agents_page_elements[1] %>%
     html_nodes('p') %>%
     html_text()
as.data.frame(read.dcf(textConnection(new)))

-输出

    Cell                            Email                                                             Professional Credentials
1 (240) 668-4241 barry.aberdeen@tributesports.com NBA Certified Player Agent, FIBA Certified Player Agent, WNBA Certified Player Agent

---

对于多个元素，使用map

library(purrr)
library(dplyr)
library(stringr)
out <- map_dfr(agents_page_elements, ~ {
       new <- .x %>% 
               html_nodes('p') %>%
               html_text() %>%
               str_replace_all("\n\\s*", " ")
    if(length(new) > 0) {
       as.data.frame(read.dcf(textConnection(new)))
     } else NULL

     })

-输出

> dim(out)
[1] 455   9
> head(out, 2)
            Cell                            Email                                                             Professional Credentials    Title
1 (240) 668-4241 barry.aberdeen@tributesports.com NBA Certified Player Agent, FIBA Certified Player Agent, WNBA Certified Player Agent     <NA>
2 (281) 773-7339                             <NA>                                     Texas Bar No. 24050197|Wisconsin Bar No. 1045470 Attorney
             Company Name                                                                                             Education Address Office International
1                    <NA>                                                                                                  <NA>    <NA>   <NA>          <NA>
2 Adams & Associates, LLC University of Southern California B.S. |University of Houston - M.B.A |University of Wisconsin - J.D.    <NA>   <NA>          <NA>