使用 pandas 比较具有不同列数的大型 CSV 文件答案

【问题标题】：using pandas to compare large CSV files with different numbers of columns使用 pandas 比较具有不同列数的大型 CSV 文件
【发布时间】：2018-02-25 17:35:47
【问题描述】：

我是 python 编程的新手，我正在尝试加入两个具有不同列数的 csv 文件。目的是查找缺失的记录并创建包含主列中特定列的报告。

直接从excel复制的两个csv文件的例子 CSV 样本 1(combine201709.csv)

start_time  end_time    aitechid    hh_village  grpdetails1/farmername  grpdetails1/farmermobile
2016-11-26T14:01:47.329+03  2016-11-26T14:29:05.042+03  AI00001 2447    KahsuGebru  919115604
2016-11-26T19:34:42.159+03  2016-11-26T20:39:27.430+03  936891238   2473    Moto Aleka  914370833
2016-11-26T12:13:23.094+03  2016-11-26T14:25:19.178+03  914127382   2390    Hagos   914039654
2016-11-30T14:31:28.223+03  2016-11-30T14:56:33.144+03  920784222

SAMPLE CSV 2 (combinedmissingrecords.csv)

farmermobile
941807851
946741296
9
920212218
915
939555303
961579437
919961811
100004123
972635273
918166831
961579437
922882638
100006273
919728710
30000739
920770648
100004727
963767487
915855665
932255143
923531603
0
931875236
918027506
8
916353266
918020303
924359729
934623027
916585963
960791618
988047183
100002632
300007241
918271897
300007238
918250712

我试过了，但无法获得预期的输出：

    import pandas as pd

normalize = lambda x: "%.4f" % float(x) # round
df = pd.read_csv("/media/dmogaka/DATA/week progress/week4/combine201709.csv", index_col=(0,1), usecols=(1, 2, 3,4),
                 header=None, converters=dict.fromkeys([1,2]))
df2 = pd.read_csv("/media/dmogaka/DATA/week progress/week4/combinedmissingrecords.csv", index_col=(0,1), usecols=(0),
                  header=None, converters=dict.fromkeys([1,2]))
result = df2.merge(df[['aitechid','grpdetails1/farmermobile','grpdetails1/farmername']],
         left_on='farmermobile', right_on='grpdetails1/farmermobile')
result.to_csv("/media/dmogaka/DATA/week progress/week4/output.csv", header=None) # write as csv

错误信息

/usr/bin/python3.5 "/media/dmogaka/DATA/Panda tut/test/test.py"
Traceback (most recent call last):
  File "/media/dmogaka/DATA/Panda tut/test/test.py", line 7, in <module>
    header=None, converters=dict.fromkeys([1,2]))
  File "/home/dmogaka/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 655, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/dmogaka/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 405, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/dmogaka/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 764, in __init__
    self._make_engine(self.engine)
  File "/home/dmogaka/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 985, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/home/dmogaka/.local/lib/python3.5/site-packages/pandas/io/parsers.py", line 1605, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 461, in pandas._libs.parsers.TextReader.__cinit__ (pandas/_libs/parsers.c:4968)
TypeError: 'int' object is not iterable

Process finished with exit code 1

【问题讨论】：

Comparing two pandas dataframes for differences的可能重复
@MrE，我不认为这是重复的。如果我们有不同的列数 assert_frame_equal 将始终返回 AssertionError
您能否发布两个小的（3-5 行）样本可重现数据集和您想要的结果数据集？请阅读how to make good reproducible pandas examples并相应地编辑您的帖子。
只是猜测您是否要比较数据帧，它们应该具有相同的格式。所以第一步是修剪/调整格式以获得可比较的 DF，然后根据其他帖子进行比较
@MrE，假设我们想查看第一个 DF 中缺少哪些行，而第二个 DF 中存在哪些行...

标签： python pandas csv

【解决方案1】：

试试这个：

d2.merge(d1[['aitechid','grpdetails1/farmermobile','grpdetails1/farmername']], 
         left_on='farmermobile', right_on='grpdetails1/farmermobile')

或

d2.merge(d1[['aitechid','grpdetails1/farmermobile','grpdetails1/farmername']] \
          .rename(columns={'grpdetails1/farmermobile':'farmermobile'}))

【讨论】：

我已经尝试过您的代码，但我不断收到@MaxU 上方的错误消息