将单列数据聚合到数据框中答案

【问题标题】：aggregating single column data into a dataframe将单列数据聚合到数据框中
【发布时间】：2021-04-27 20:03:40
【问题描述】：

我有一个像这样的具有相同键值的表

Measure	element	unit	value
Measurment N° 000001 - A	NA	NA	NA
Point 0000000101	NA	NA	NA
Station alpha	NA	NA	NA
NA	NA	NA	11-Jan-2018
NA	Parameter A	mg	5
NA	Parameter C	mg	6
NA	Parameter E	mg	8
NA	Parameter F	mg	3
NA	Parameter G	mg	4
Measurment N° 000003 - A	NA	NA	NA
Point 0000000121	NA	NA	NA
Station bravo	NA	NA	NA
NA	NA	NA	19-Jun-2019
NA	Parameter A	mg	1
NA	Parameter B	mg	9
NA	Parameter D	g	5
NA	Parameter F	mg	6
NA	Parameter G	mg	3
Measurment N° 000003 - B	NA	NA	NA
Point 0000000122	NA	NA	NA
Station charlie	NA	NA	NA
NA	NA	NA	17-Jan-2020
NA	Parameter A	mg	9
NA	Parameter E	mg	5
NA	Parameter F	mg	3

df <- as_tibble(list(measure = c('Measurment N° 000001 - A', 'Point 0000000101', 'Station alpha', NA, NA, NA, NA, NA, NA,'Measurment N° 000003 - A', 'Point 0000000121', 'Station bravo', NA, NA, NA, NA, NA, NA,'Measurment N° 000003 - B', 'Point 0000000122', 'Station Charlie', NA, NA, NA, NA),
                      element = c(NA, NA, NA, NA, 'Parameter A', 'Parameter C','Parameter E','Parameter F','Parameter G',NA, NA, NA, NA, 'Parameter A', 'Parameter B','Parameter D','Parameter F','Parameter G',NA, NA, NA, NA, 'Parameter A', 'Parameter E','Parameter F'),
                      unit = c(NA, NA , NA, NA, 'mg', 'mg', 'mg', 'mg', 'mg',NA, NA , NA, NA, 'mg', 'mg', 'g', 'mg', 'mg',NA, NA , NA, NA, 'mg', 'mg', 'mg'),
                      value = c(NA, NA , NA, '11-Jan-2018', 5, 6, 8, 3, 4,NA, NA , NA, '19-Jun-2019', 1, 9, 5, 6, 3,NA, NA , NA, '17-Jan-2020', 9, 5, 3)))

我必须按点获取数据，测量站是这样的：

Measurement	point	station	date	Parameter A	Parameter B	Parameter C	Parameter D	Parameter E	Parameter F	Parameter G
NA	NA	NA	NA	mg	mg	mg	g	mg	mg	mg
Measurment N° 000001 - A	Point 0000000101	Station alpha	11-Jan-2018	5	NA	6	NA	8	3	4
Measurment N° 000003 - A	Point 0000000121	Station bravo	19-Jun-2019	1	9	NA	5	NA	6	3
Measurment N° 000003 - B	Point 0000000122	Station Charlie	17-Jan-2020	9	NA	NA	NA	5	3	NA

我正在寻找一种使用 R 或 pandas 聚合这些数据的方法

【问题讨论】：

通过将"mg" 预期为框架中的行，您可以保证您的所有数字都是字符串，而不是数字。 R 不像其他工具那样做两个标题行。我建议将数字存储为数字，或者对列名中的单位进行编码，或者如果它们不完全相同，则将它们存储在另一组列中。
我可以放弃那个单位线，这并不重要

标签： python r pandas dataframe dplyr

【解决方案1】：

我将使用一组混合的基本/tidyverse 代码。

基础部分用于切出每组行。我们可以 grep 查找 ^Meas 并根据它分配组。

cumsum(grepl("^Meas", dat$Measure))
#  [1] 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3

func <- function(x) {
  vals <- na.omit(unlist(x[1:4,c("Measure", "value")]))
  stopifnot(length(vals) == 4L)
  left <- setNames(
    as.data.frame(matrix(vals, nrow = 1)),
    c("Measure", "point", "station", "date"))
  right <- as.data.frame(matrix(x$value[-(1:4)], nrow = 1, dimnames = list(NULL, x$element[-(1:4)])))
  cbind(left, right)
}

func(dat[1:9,])
#                    Measure            point       station        date Parameter A Parameter C Parameter E Parameter F Parameter G
# 1 Measurment N° 000001 - A Point 0000000101 Station alpha 11-Jan-2018           5           6           8           3           4

从这里开始，它是分组和组合。虽然我们可以使用 do.call(rbind, ..) 对帧列表进行逐行组合，但当元素不具有相同的列和顺序时，它需要更多的工作。

library(dplyr)
dat %>%
  group_by(grp = cumsum(grepl("^Meas", Measure))) %>%
  do(func(.)) %>%
  ungroup() %>%
  select(Measure, point, station, date, sort(colnames(.)), -grp)
# # A tibble: 3 x 11
#   Measure                  point            station         date        `Parameter A` `Parameter B` `Parameter C` `Parameter D` `Parameter E` `Parameter F` `Parameter G`
#   <chr>                    <chr>            <chr>           <chr>       <chr>         <chr>         <chr>         <chr>         <chr>         <chr>         <chr>        
# 1 Measurment N° 000001 - A Point 0000000101 Station alpha   11-Jan-2018 5             <NA>          6             <NA>          8             3             4            
# 2 Measurment N° 000003 - A Point 0000000121 Station bravo   19-Jun-2019 1             9             <NA>          5             <NA>          6             3            
# 3 Measurment N° 000003 - B Point 0000000122 Station charlie 17-Jan-2020 9             <NA>          <NA>          <NA>          5             3             <NA>

数据

dat <- structure(list(Measure = c("Measurment N° 000001 - A", "Point 0000000101", "Station alpha", NA, NA, NA, NA, NA, NA, "Measurment N° 000003 - A", "Point 0000000121", "Station bravo", NA, NA, NA, NA, NA, NA, "Measurment N° 000003 - B", "Point 0000000122", "Station charlie", NA, NA, NA, NA), element = c(NA, NA, NA, NA, "Parameter A", "Parameter C", "Parameter E", "Parameter F", "Parameter G", NA, NA, NA, NA, "Parameter A", "Parameter B", "Parameter D", "Parameter F", "Parameter G", NA, NA, NA, NA, "Parameter A", "Parameter E", "Parameter F"), unit = c(NA, NA, NA, NA, "mg", "mg", "mg", "mg", "mg", NA, NA, NA, NA, "mg", "mg", "g", "mg", "mg", NA, NA, NA, NA, "mg", "mg", "mg"), value = c(NA, NA, NA, "11-Jan-2018", "5", "6", "8", "3", "4", NA, NA, NA, "19-Jun-2019", "1", "9", "5", "6", "3", NA, NA, NA, "17-Jan-2020", "9", "5", "3")), class = "data.frame", row.names = c(NA, -25L))
expected <- structure(list(Measurement = c("Measurment N° 000001 - A", "Measurment N° 000003 - A", "Measurment N° 000003 - B"), point = c("Point 0000000101", "Point 0000000121", "Point 0000000122"), station = c("Station alpha", "Station bravo", "Station Charlie"), date = c("11-Jan-2018", "19-Jun-2019", "17-Jan-2020"), Parameter.A = c(5L, 1L, 9L), Parameter.B = c(NA, 9L, NA),     Parameter.C = c(6L, NA, NA), Parameter.D = c(NA, 5L, NA),     Parameter.E = c(8L, NA, 5L), Parameter.F = c(3L, 6L, 3L),     Parameter.G = c(4L, 3L, NA)), class = "data.frame", row.names = c(NA, -3L))

【讨论】：

【解决方案2】：

有点粗略，但很有效

library(tidyverse)

df %>% select(-unit) %>%
  mutate(element = case_when(str_detect(Measure, "Measurment") | 
                           str_detect(Measure, "Point") |
                           str_detect(Measure, "Station") ~ Measure, 
                         TRUE ~ element),
         Measure = case_when(str_detect(Measure, "Measurment") ~ Measure,
                              TRUE ~ NA_character_)) %>%
  fill(Measure) %>% mutate(element = ifelse(is.na(element), "date", element)) %>%
  filter(!str_detect(element, "Measurment")) %>%
  mutate(value = ifelse(is.na(value), element, value),
         element = case_when(str_detect(element, "Point") ~ "Point",
                             str_detect(element, "Station") ~ "Station",
                             TRUE ~ element)) %>%
  pivot_wider(names_from = element, values_from = value)

# A tibble: 3 x 11
  Measure Point Station date  `Parameter A` `Parameter C` `Parameter E` `Parameter F` `Parameter G`
  <chr>   <chr> <chr>   <chr> <chr>         <chr>         <chr>         <chr>         <chr>        
1 Measur~ Poin~ Statio~ 11-J~ 5             6             8             3             4            
2 Measur~ Poin~ Statio~ 19-J~ 1             NA            NA            6             3            
3 Measur~ Poin~ Statio~ 17-J~ 9             NA            5             3             NA           
# ... with 2 more variables: `Parameter B` <chr>, `Parameter D` <chr>

样本输入

> dput(df)
structure(list(Measure = c("Measurment N° 000001 - A", "Point 0000000101", 
"Station alpha", NA, NA, NA, NA, NA, NA, "Measurment N° 000003 - A", 
"Point 0000000121", "Station bravo", NA, NA, NA, NA, NA, NA, 
"Measurment N° 000003 - B", "Point 0000000122", "Station charlie", 
NA, NA, NA, NA), element = c(NA, NA, NA, NA, "Parameter A", "Parameter C", 
"Parameter E", "Parameter F", "Parameter G", NA, NA, NA, NA, 
"Parameter A", "Parameter B", "Parameter D", "Parameter F", "Parameter G", 
NA, NA, NA, NA, "Parameter A", "Parameter E", "Parameter F"), 
    unit = c(NA, NA, NA, NA, "mg", "mg", "mg", "mg", "mg", NA, 
    NA, NA, NA, "mg", "mg", "g", "mg", "mg", NA, NA, NA, NA, 
    "mg", "mg", "mg"), value = c(NA, NA, NA, "11-Jan-18", "5", 
    "6", "8", "3", "4", NA, NA, NA, "19-Jun-19", "1", "9", "5", 
    "6", "3", NA, NA, NA, "17-Jan-20", "9", "5", "3")), class = "data.frame", row.names = c(NA, 
-25L))

【讨论】：

【解决方案3】：

使用 pandas，您可以通过标记以 'Measurment' 开头的行来为数据块创建唯一标识符。 pivot 将所有参数转换为每个块的宽格式，groupby 会将测量/日期信息拆分为每个块的单独字段。然后我们将所有数据连接在一起。

这些单位会按照您指定的方式导致 dtypes 出现问题，因此要么不包含它们，要么在第一块代码之后查看替代方案，这将使 Series 保持为数字 dtypes。

df['idx'] = df['measure'].str.startswith('Measurment', na=False).astype(int).cumsum()

params = (df[df.element.notnull()].pivot(index='idx', columns='element', values='value')
            .rename_axis(columns=None, index=None))
units = df.groupby('element').unit.first()

# Add a row of units at the top, or ignore this completely. 
params = pd.concat([pd.DataFrame([params.columns.map(units)], columns=params.columns, index=[0]),
                    params])

info = df.groupby('idx').agg(Measurement=('measure', 'first'),
                             point=('measure', lambda x: x.iloc[1]),
                             station=('measure', lambda x: x.iloc[2]),
                             date=('value', 'first'))

result = pd.concat([info, params], axis=1)

                Measurement             point          station         date Parameter A Parameter B Parameter C Parameter D Parameter E Parameter F Parameter G
0                       NaN               NaN              NaN          NaN          mg          mg          mg           g          mg          mg          mg
1  Measurment N° 000001 - A  Point 0000000101    Station alpha  11-Jan-2018           5         NaN           6         NaN           8           3           4
2  Measurment N° 000003 - A  Point 0000000121    Station bravo  19-Jun-2019           1           9         NaN           5         NaN           6           3
3  Measurment N° 000003 - B  Point 0000000122  Station Charlie  17-Jan-2020           9         NaN         NaN         NaN           5           3         NaN

但正如 cmets 中所指出的，要求单位为这些系列中的值将创建 object 列，这使得后续计算变得困难。我们可以改为使用MultiIndex，concat 会将其展平为一个元组。所以而不是"Add a row of units at top" 使用的那行代码

params.columns = pd.MultiIndex.from_arrays([params.columns, params.columns.map(units)])

在其余代码之后的结果是：

                Measurement             point          station         date (Parameter A, mg) (Parameter B, mg) (Parameter C, mg) (Parameter D, g) (Parameter E, mg) (Parameter F, mg) (Parameter G, mg)
1  Measurment N° 000001 - A  Point 0000000101    Station alpha  11-Jan-2018                 5               NaN                 6              NaN                 8                 3                 4
2  Measurment N° 000003 - A  Point 0000000121    Station bravo  19-Jun-2019                 1                 9               NaN                5               NaN                 6                 3
3  Measurment N° 000003 - B  Point 0000000122  Station Charlie  17-Jan-2020                 9               NaN               NaN              NaN                 5                 3               NaN

【讨论】：

在我的文件中有很多数据，在参数中我遇到了由于具有相同名称的测量结果而具有相同索引的问题，因此在旋转它时返回'索引包含重复条目，不能重塑'
我具有唯一值的列是“点”列