【问题标题】:Using loop to extract data from list in R使用循环从R中的列表中提取数据
【发布时间】:2018-02-15 04:20:04
【问题描述】:

我是 R 的新手。我昨天抓取了一个需要登录的网站,页面是 xml 格式,如下所示。

<result status="success">
  <code>1</code>
  <note>success</note>
  <teacherList>
    <teacher id="D95">
      <name>Mary</name>
      <department id="420">
        <name>Math</name>
      </department>
      <department id="421">
        <name>Statistics</name>
      </department>
    </teacher>
    <teacher id="D73">
      <name>Adam</name>
      <department id="412">
        <name>English</name>
      </department>
    </teacher>
  </teacherList>
</result> 

最近我刚刚将 XML 转换为列表。

library(XML)
library(rvest)
library(plyr)
library(dplyr)
library(httr)
library(pipeR)
library(xml2)

url.address <- "http://xxxxxxxxxxxxxxxxx"
session <-html_session(url.address)
form <-html_form(read_html(url.address))[[1]]
filled_form <- set_values(form,
                          "userid" = "id",
                          "Password" = "password")
s <- submit_form(session,filled_form)
z = read_xml(s$response)
z1 = as_list(z)
z2 <- z1$teacherList

现在我需要从列表中提取数据并将其作为数据框。顺便说一句,有些人属于 2 个部门,但有些人只属于 1 个部门。列表 z2 的一部分如下所示:

z2[[1]]

$name
$name[[1]]
[1] "Mary"


$department
$department$name
$department$name[[1]]
[1] "Math"


attr(,"id")
[1] "420"

$department
$department$name
$department$name[[1]]
[1] "statistics"


attr(,"id")
[1] "421"

attr(,"id")
[1] "D95236"

当我一一提取时,时间太长了:

attr(z2[[1]],"id")

“D95”

z2[[1]][[1]][[1]] 

“玛丽”

z2[[1]][[2]][[1]][[1]] 

“数学”

attr(z2[[1]][[2]], "id") 

“420”

z2[[1]][[3]][[1]][[1]] 

“统计”

attr(z2[[1]][[3]], "id")

“421”

attr(z2[[2]],"id")

“D73”

z2[[2]][[1]][[1]] 

“亚当”

z2[[2]][[2]][[1]][[1]]

“英语”

attr(z2[[2]][[2]],"id")

“412”

于是我试着写了一个循环:

for (x in 1:2){
  for (y in 2:3){
  a <- attr(z2[[x]],"id")
  b <- z2[[x]][[1]][[1]]
  d <- z2[[x]][[y]][[1]][[1]]
  e <- attr(z2[[x]][[y]],"id")
  g <- cbind(print(a),print(b),print(d),print(e))
  }}

但它根本不起作用,因为有些人只属于一个部门。我预期的结果:

任何建议将不胜感激!

dput(head(z2, 10))

structure(list(teacher = structure(list(name = list("Mary"), 
    department = structure(list(name = list("Math")), .Names = "name", id = "420"), 
    department = structure(list(name = list("statistics")), .Names = "name", id = "421")), .Names = c("name", 
"department", "department"), id = "D95"), teacher = structure(list(
    name = list("Adam"), department = structure(list(name = list(
        "English")), .Names = "name", id = "412")), .Names = c("name", 
"department"), id = "D73"), teacher = structure(list(name = list(
    "Kevin"), department = structure(list(name = list("Chinese")), .Names = "name", id = "201")), .Names = c("name", 
"department"), id = "D101"), teacher = structure(list(name = list(
    "Nana"), department = structure(list(name = list("Science")), .Names = "name", id = "205")), .Names = c("name", 
"department"), id = "D58"), teacher = structure(list(name = list(
    "Nelson"), department = structure(list(name = list("Music")), .Names = "name", id = "370")), .Names = c("name", 
"department"), id = "D14"), teacher = structure(list(name = list(
    "Esther"), department = structure(list(name = list("Medicine")), .Names = "name", id = "361")), .Names = c("name", 
"department"), id = "D28"), teacher = structure(list(name = list(
    "Mia"), department = structure(list(name = list("Chemistry")), .Names = "name", id = "326")), .Names = c("name", 
"department"), id = "D17"), teacher = structure(list(name = list(
    "Jack"), department = structure(list(name = list("German")), .Names = "name", id = "306")), .Names = c("name", 
"department"), id = "D80"), teacher = structure(list(name = list(
    "Tom"), department = structure(list(name = list("French")), .Names = "name", id = "360")), .Names = c("name", 
"department"), id = "D53"), teacher = structure(list(name = list(
    "Allen"), department = structure(list(name = list("Spanish")), .Names = "name", id = "322")), .Names = c("name", 
"department"), id = "D18")), .Names = c("teacher", "teacher", 
"teacher", "teacher", "teacher", "teacher", "teacher", "teacher", "teacher", 
"teacher"))

【问题讨论】:

  • 除非您提供可重现的数据示例,否则无法提供帮助。尝试dput(head(z2, 10)) 并将结果粘贴到您的问题中。
  • @lmo 对不起!刚刚添加:)
  • 请不要粘贴代码图片。请阅读how to make a great reproducible example
  • @lmo 现在就上传吧。对不起,我还没有弄清楚如何发布输出,所以我上传了图片。很抱歉给您带来不便。
  • @C8H10N4O2 嗨!非常抱歉,前两天才开始使用。我知道这不应该成为我的借口。我会尽快弄清楚如何做。

标签: r xml list loops extract


【解决方案1】:

构建起来有点疯狂,但我认为它或多或少符合帖子先前版本中发布的所需输出。我不得不在lapply 函数中使用sapply 来提取第二个ID 变量。

do.call(rbind,             # rbind list of data.frames output by lapply
        lapply(unname(z2), # loop through list, first drop outer names
               function(x) { # begin lapply function
                 temp <- unlist(x) # unlist inner elements to a vector
                 data.frame(name=temp[names(temp) == "name"], # subset on names
                            dept=temp[names(temp) == "department.name"], # subset on dept
                            id=attr(x, "id"), # extract one id
                            id2=unlist(sapply(x, attr, "id")), # extract other id
                            row.names=NULL) # end data.frame function, drop row.names
                            })) # end lapply function, lapply, and do.call

返回

     name       dept   id id2
1    Mary       Math  D95 420
2    Mary statistics  D95 421
3    Adam    English  D73 412
4   Kevin    Chinese D101 201
5    Nana    Science  D58 205
6  Nelson      Music  D14 370
7  Esther   Medicine  D28 361
8     Mia  Chemistry  D17 326
9    Jack     German  D80 306
10    Tom     French  D53 360
11  Allen    Spanish  D18 322

第二个列表的结构在许多方面与最初的示例不同。第一:一个巢被移除。也就是说,新列表的深度比初始示例的深度小一。就好像您为初始列表提供了 z2[[1]] 。其次,第二个示例缺少我最初称为 id 的内容(例如 D95 和 D101 等值)。

通过对原始代码进行一些操作,我可以使用它

lapply(list(z3), # loop through list, first drop outer names
       function(x) { # begin lapply function
           temp <- unlist(x) # unlist inner elements to a vector
           data.frame(name=temp[names(temp) == "name"], # subset on names
                      dept=temp[names(temp) == "department.name"], # subset on dept
                      # id=attr(x, "id"), # extract one id
                      id2=unlist(sapply(x, attr, "id")), # extract other id
                      row.names=NULL) # end data.frame function, drop row.names
       })

我在z2之前提到的代码地址的更改被list(z3)替换为lapply的第一个参数,它构造了所需的列表深度。另外,内部函数id=attr(x, "id"),的行已被注释掉,因为id2不存在。

【讨论】:

  • 这很整洁!非常感谢:)
  • 我尝试使用 do.call 函数来解决另一个结构较少的问题,但它出现了错误,例如“Data.frame 中的错误(名称(临时)==“名称”,除法= temp[names(temp) == : arguments 暗示不同的行数:1, 0"。你能告诉我哪些部分出错了吗?即使只是一个提示也会很好:) 我只是在上面列出
  • @Ching 我修改了代码以使用您的第二个示例。在处理问题时,了解您的底层数据结构非常重要。在这里,当您在屏幕上打印一个小示例时,您应该已经看到该数据中缺少 id 变量。
  • 嗨!当我使用 list(z3) 时它不起作用,但是当我使用 unname(z3) 时它又起作用了。这很奇怪。我确实注意到了结构的不同。
  • 等等!我知道为什么它不起作用,我上面列出的第二个结构是错误的(我之前没有注意到)。太感谢了!你真的救了我。这种结构真的让我抓狂:p
【解决方案2】:

XML 通常在 R 中很容易处理

使用library(XML)library(plyr) 避免编写循环:

第一步是读入XML

我将您的示例 XML 保存为名为 Demo.xml 的 .xml 文件。您还可以向 xmlParse 传递一个 URL。

rawXML &lt;- xmlParse("Demo.xml")

然后将 XML 转换为列表:

xmlList &lt;- xmlToList(rawXML)

然后使用plyr将列表转换为数据框

df1 &lt;- ldply(xmlList, data.frame)

这是一般流程,如果您提供示例数据,我们可以对其进行优化以匹配您的特定用例。

这是生成的摘要输出。这是你要找的吗?

 str(df1)
'data.frame':   4 obs. of  12 variables:
 $ .id                        : chr  "code" "note" "teacherList" ".attrs"
 $ X..i..                     : Factor w/ 2 levels "1","success": 1 2 NA 2
 $ teacher.name               : Factor w/ 1 level "Mary": NA NA 1 NA
 $ teacher.department.name    : Factor w/ 1 level "Math": NA NA 1 NA
 $ teacher.department..attrs  : Factor w/ 1 level "420": NA NA 1 NA
 $ teacher.department.name.1  : Factor w/ 1 level "Statistics": NA NA 1 NA
 $ teacher.department..attrs.1: Factor w/ 1 level "421": NA NA 1 NA
 $ teacher..attrs             : Factor w/ 1 level "D95": NA NA 1 NA
 $ teacher.name.1             : Factor w/ 1 level "Adam": NA NA 1 NA
 $ teacher.department.name.2  : Factor w/ 1 level "English": NA NA 1 NA
 $ teacher.department..attrs.2: Factor w/ 1 level "412": NA NA 1 NA
 $ teacher..attrs.1           : Factor w/ 1 level "D73": NA NA 1 NA

【讨论】:

  • 嗨!实际上我抓取了一个 xml 格式的登录网站,所以我猜它很难解析:) 我现在刚刚添加了更多信息!
  • @Ching 你能提供 XML 表格的链接吗?
  • 我觉得已经很接近了,我会尽快检查并告诉你:)
猜你喜欢
  • 2012-10-04
  • 1970-01-01
  • 2016-10-11
  • 2021-12-05
  • 2018-02-18
  • 2021-08-15
  • 1970-01-01
  • 2020-04-01
  • 1970-01-01
相关资源
最近更新 更多