如何取出这些元素并将它们组合成一个数据框答案

【问题标题】：How to take out these elements and put them together into a dataframe如何取出这些元素并将它们组合成一个数据框
【发布时间】：2020-01-17 21:41:40
【问题描述】：

[{'complete': True, 'volume': 116, 'time': '2020-01-17T19:15:00.000000000Z', 'mid': {'o': '1.10916', 'h': '1.10917', 'l': '1.10906', 'c': '1.10912'}}, {'complete': True, 'volume': 136, 'time': '2020-01-17T19:30:00.000000000Z', 'mid': {'o': '1.10914', 'h': '1.10922', 'l': '1.10908', 'c': '1.10919'}}, {'complete': True, 'volume': 223, 'time': '2020-01-17T19:45:00.000000000Z', 'mid': {'o': '1.10920', 'h': '1.10946', 'l': '1.10920', 'c': '1.10930'}}, {'complete': True, 'volume': 203, 'time': '2020-01-17T20:00:00.000000000Z', 'mid': {'o': '1.10930', 'h': '1.10931', 'l': '1.10919', 'c': '1.10928'}}, {'complete': True, 'volume': 87, 'time': '2020-01-17T20:15:00.000000000Z', 'mid': {'o': '1.10926', 'h': '1.10934', 'l': '1.10922', 'c': '1.10926'}}, {'complete': True, 'volume': 102, 'time': '2020-01-17T20:30:00.000000000Z', 'mid': {'o': '1.10926', 'h': '1.10928', 'l': '1.10913', 'c': '1.10920'}}, {'complete': True, 'volume': 277, 'time': '2020-01-17T20:45:00.000000000Z', 'mid': {'o': '1.10918', 'h': '1.10929', 'l': '1.10913', 'c': '1.10928'}}, {'complete': True, 'volume': 103, 'time': '2020-01-17T21:00:00.000000000Z', 'mid': {'o': '1.10927', 'h': '1.10929', 'l': '1.10920', 'c': '1.10924'}}, {'complete': True, 'volume': 54, 'time': '2020-01-17T21:15:00.000000000Z', 'mid': {'o': '1.10926', 'h': '1.10926', 'l': '1.10910', 'c': '1.10912'}}, {'complete': False, 'volume': 15, 'time': '2020-01-17T21:30:00.000000000Z', 'mid': {'o': '1.10913', 'h': '1.10918', 'l': '1.10912', 'c': '1.10913'}}]

我尝试从这个列表中去掉所有的“时间”和“中间”。在'mid'中，有'o'、'h'、'l'、'c'字典。有没有办法将“时间”和这些字典组合成一个数据框？

【问题讨论】：

标签： python python-3.x dataframe dictionary data-cleaning

【解决方案1】：

试试

df = pd.DataFrame(your_data)
df = pd.concat([df['time'], df['mid'].apply(pd.Series)], axis=1)

                             time        o        h        l        c
0  2020-01-17T19:15:00.000000000Z  1.10916  1.10917  1.10906  1.10912
1  2020-01-17T19:30:00.000000000Z  1.10914  1.10922  1.10908  1.10919
2  2020-01-17T19:45:00.000000000Z  1.10920  1.10946  1.10920  1.10930
3  2020-01-17T20:00:00.000000000Z  1.10930  1.10931  1.10919  1.10928
4  2020-01-17T20:15:00.000000000Z  1.10926  1.10934  1.10922  1.10926
5  2020-01-17T20:30:00.000000000Z  1.10926  1.10928  1.10913  1.10920
6  2020-01-17T20:45:00.000000000Z  1.10918  1.10929  1.10913  1.10928
7  2020-01-17T21:00:00.000000000Z  1.10927  1.10929  1.10920  1.10924
8  2020-01-17T21:15:00.000000000Z  1.10926  1.10926  1.10910  1.10912
9  2020-01-17T21:30:00.000000000Z  1.10913  1.10918  1.10912  1.10913

【讨论】：

【解决方案2】：

尝试以下方法：

import pandas as pd

l = [{'complete': True, 'volume': 116, 'time': '2020-01-17T19:15:00.000000000Z', 'mid': {'o': '1.10916', 'h': '1.10917', 'l': '1.10906', 'c': '1.10912'}}, {'complete': True, 'volume': 136, 'time': '2020-01-17T19:30:00.000000000Z', 'mid': {'o': '1.10914', 'h': '1.10922', 'l': '1.10908', 'c': '1.10919'}}, {'complete': True, 'volume': 223, 'time': '2020-01-17T19:45:00.000000000Z', 'mid': {'o': '1.10920', 'h': '1.10946', 'l': '1.10920', 'c': '1.10930'}}, {'complete': True, 'volume': 203, 'time': '2020-01-17T20:00:00.000000000Z', 'mid': {'o': '1.10930', 'h': '1.10931', 'l': '1.10919', 'c': '1.10928'}}, {'complete': True, 'volume': 87, 'time': '2020-01-17T20:15:00.000000000Z', 'mid': {'o': '1.10926', 'h': '1.10934', 'l': '1.10922', 'c': '1.10926'}}, {'complete': True, 'volume': 102, 'time': '2020-01-17T20:30:00.000000000Z', 'mid': {'o': '1.10926', 'h': '1.10928', 'l': '1.10913', 'c': '1.10920'}}, {'complete': True, 'volume': 277, 'time': '2020-01-17T20:45:00.000000000Z', 'mid': {'o': '1.10918', 'h': '1.10929', 'l': '1.10913', 'c': '1.10928'}}, {'complete': True, 'volume': 103, 'time': '2020-01-17T21:00:00.000000000Z', 'mid': {'o': '1.10927', 'h': '1.10929', 'l': '1.10920', 'c': '1.10924'}}, {'complete': True, 'volume': 54, 'time': '2020-01-17T21:15:00.000000000Z', 'mid': {'o': '1.10926', 'h': '1.10926', 'l': '1.10910', 'c': '1.10912'}}, {'complete': False, 'volume': 15, 'time': '2020-01-17T21:30:00.000000000Z', 'mid': {'o': '1.10913', 'h': '1.10918', 'l': '1.10912', 'c': '1.10913'}}]

df = pd.DataFrame()

for ll in l:
     df = df.append(pd.DataFrame(ll['mid'], index=[ll['time']]))

【讨论】：

【解决方案3】：

假设您的样本数据名为data：

>>> pd.DataFrame([d['mid'] for d in data], index=[d['time'] for d in data])
                                      o        h        l        c
2020-01-17T19:15:00.000000000Z  1.10916  1.10917  1.10906  1.10912
2020-01-17T19:30:00.000000000Z  1.10914  1.10922  1.10908  1.10919
2020-01-17T19:45:00.000000000Z  1.10920  1.10946  1.10920  1.10930
2020-01-17T20:00:00.000000000Z  1.10930  1.10931  1.10919  1.10928
2020-01-17T20:15:00.000000000Z  1.10926  1.10934  1.10922  1.10926
2020-01-17T20:30:00.000000000Z  1.10926  1.10928  1.10913  1.10920
2020-01-17T20:45:00.000000000Z  1.10918  1.10929  1.10913  1.10928
2020-01-17T21:00:00.000000000Z  1.10927  1.10929  1.10920  1.10924
2020-01-17T21:15:00.000000000Z  1.10926  1.10926  1.10910  1.10912
2020-01-17T21:30:00.000000000Z  1.10913  1.10918  1.10912  1.10913

时间

data *= 1000  # Now list of 10k dictionaries.

%timeit df = pd.DataFrame([d['mid'] for d in data], index=[d['time'] for d in data])
# 13.4 ms ± 361 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
df = pd.DataFrame(data)
df = pd.concat([df['time'], df['mid'].apply(pd.Series)], axis=1)
# 4.52 s ± 494 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
df = pd.DataFrame()
for record in data:
     df = df.append(pd.DataFrame(record['mid'], index=[record['time']]))
# 21.4 s ± 2.86 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

【讨论】：

我很好奇 for 循环方法对于大数据是否会变得非常低效？
不，不在这个上下文中。
真的吗？嗯...您不会在高层逐行阅读data，而且解释器还必须在每次迭代中确定d 的dtype。在这种情况下如何被忽略？
从时间安排上可以看出，我的方法胜过其他两种方法。我最初用 100 万行制作了 data，而我的方法只花了一秒钟多一点。 20 分钟后，当其他两种方法中更快的方法仍未完成时，我不得不杀死它。
Alexander 的方法具有更好的性能，但是您需要在创建数据帧之前将完整的数据集加载到内存中。我相信内存在这里不是问题，所以除非你的系统非常有限，否则这是最好的选择。