【问题标题】:How can I combine multiple CSV files that share an identical column into a new CSV file?如何将共享同一列的多个 CSV 文件合并到一个新的 CSV 文件中?
【发布时间】:2020-01-19 00:09:38
【问题描述】:

所以我有多个 csv 数据文件,我想将它们放入一个主文件中。每个 CSV 文件有两列,第一列有标记的 bin,第二列表示落在这些 bin 中的数据点的计数。我想以一种使我能够制作热图或 2d 直方图的方式组合每个 csv 文件,我关注了this tutorial,但它没有达到预期的效果。输出的第一行最终看起来像这样,只是前几行

Coordinate Bins,Counts for time interval 0,Coordinate Bins,Counts for time interval 1,Coordinate Bins,Counts for time interval 2,Coordinate Bins,Counts for time interval 3,Coordinate Bins,Counts for time interval 4,Coordinate Bins,Counts for time interval 5,Coordinate Bins,Counts for time interval 6,Coordinate Bins,Counts for time interval 7,Coordinate Bins,Counts for time interval 8,Coordinate Bins,Counts for time interval 9
"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0,"(-10, -9]",0
"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0,"(-9, -8]",0
"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0,"(-8, -7]",0
"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0,"(-7, -6]",0
"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0,"(-6, -5]",0
"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0,"(-5, -4]",0
"(-4, -3]",0,"(-4, -3]",41,"(-4, -3]",6,"(-4, -3]",20,"(-4, -3]",0,"(-4, -3]",0,"(-4, -3]",0,"(-4, -3]",5,"(-4, -3]",2,"(-4, -3]",1
"(-3, -2]",21,"(-3, -2]",52,"(-3, -2]",38,"(-3, -2]",52,"(-3, -2]",1,"(-3, -2]",0,"(-3, -2]",0,"(-3, -2]",68,"(-3, -2]",22,"(-3, -2]",4
"(-2, -1]",13,"(-2, -1]",80,"(-2, -1]",120,"(-2, -1]",51,"(-2, -1]",34,"(-2, -1]",36,"(-2, -1]",4,"(-2, -1]",81,"(-2, -1]",40,"(-2, -1]",8

我似乎无法弄清楚如何让它只添加标签列一次而不是每次。

这里是我尝试执行此操作的 CSV 文件之一的示例,同样只是前几行

Coordinate Bins,Counts for time interval 0
"(-10, -9]",0
"(-9, -8]",0
"(-8, -7]",0
"(-7, -6]",0
"(-6, -5]",0
"(-5, -4]",0
"(-4, -3]",0
"(-3, -2]",24
"(-2, -1]",67
"(-1, 0]",126
"(0, 1]",171
"(1, 2]",339

最后是我正在使用的代码,

def combine_to_master(coordmastername, csvdir):
    os.chdir(csvdir)
    ext = "csv"
    all_filenames = [i for i in glob.glob('*.{}'.format(ext))]
    combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames], sort=True, axis=1)
    combined_csv.to_csv(coordmastername + ".csv", index=False, encoding="utf-8")

感谢您的任何帮助。

【问题讨论】:

  • 你想要什么输出? “组合”还不够:有无数种方法可以“组合”数据。是否要对每个相应的坐标 bin 求和?
  • 您在文件中是否有不同的名称Counts for time interval 0Counts for time interval 1,或者您在所有文件中都有相同的名称Counts?对于不同的名称可以工作df = pd.merge(df1, df2)。对于相同的名称,您可能需要df['Counts 0'] = df1['Counts'] df['Counts 1'] = df2['Counts']
  • 您只想要一个 long csv - 有两列,每个文件都附加到前一个文件的末尾?
  • 我建议您从一个 bin 开始,并以 2 或 3 个文件作为您问题的示例。保持简短。目前还不清楚你想要什么作为最终产品......
  • 不要试图用列表理解来做。使用 regular for 循环,您将拥有更多控制权。文件有多大?每个文件是否有相同的bins

标签: python csv


【解决方案1】:

您可以获取第一个数据帧,然后与新的数据帧合并

import pandas as pd

df1 = pd.DataFrame({'Bin': [1,2,3], 'Counts 1': [4,5,6]})
df2 = pd.DataFrame({'Bin': [1,2,3], 'Counts 2': [7,8,9]})
df3 = pd.DataFrame({'Bin': [1,2,3], 'Counts 3': [1,2,3]})

df = df1

for new_df in [df2, df3]:
    df = pd.merge(df, new_df)

print(df)

结果

   Bin  Counts 1  Counts 2  Counts 3
0    1         4         7         1
1    2         5         8         2
2    3         6         9         3

【讨论】:

  • 好吧,这似乎应该可以工作,但它会引发错误。 pandas.errors.MergeError:没有要执行合并的通用列。合并选项:left_on=None、right_on=None、left_index=False、right_index=False。我当时看了看文档,我不确定。这是否意味着我编写 csv 文件的方式可能有问题?
  • 您似乎在所有文件中都没有Coordinate Bins,或者您以错误的方式阅读它并将Coordinate Bins,Counts for time interval 0 创建为单列。您应该检查 print( df.columns )print( df.dtypes ) 的所有数据帧。
  • 我尝试将 'left_on' 和 'left_index' 指定为 '"Coordinate bins"' 并且它抛出了一个新错误 'TypeError: object of type 'NoneType' 来自 pandas 库的深处,但是起源于我称之为合并的这条线'combined_csv = pd.merge(combined_csv, pd.read_csv(csv_to_merge), left_on="Coordinate Bins", left_index=True)'
  • 正如我所说的 - 可能您以错误的方式阅读它,它不会创建列 "Coordinate bins",而是将所有数据视为具有长名称 "Coordinate Bins,Counts for time interval 0" 的列。首先,您必须在阅读文件后检查您拥有的内容 - print( df.columns )print( df.dtypes )
  • 您是否直接分配了第一个数据帧 - 请参阅我的代码中的 df = df1。它不会合并到空数据框
猜你喜欢
  • 1970-01-01
  • 2020-06-28
  • 1970-01-01
  • 2014-05-17
  • 2019-10-11
  • 2018-03-30
  • 1970-01-01
  • 2021-03-13
  • 2019-08-26
相关资源
最近更新 更多