在 Python 中比较黄金标准 csv 文件和提取值 csv 文件

Question

这适用于数据挖掘任务，我们在其中自动对提取质量进行评分。 有一个黄金标准 csv，它可能包含看起来像的字段

golden_standard.csv

| id | description             | amount  | date       |
|----|-------------------------|---------|------------|
| 1  | Some description.       | $150.54 | 12/12/2012 |
| 2  | Some other description. | $200    | 10/10/2015 |
| 3  | Other description.      | $25     | 11/11/2014 |
| 4  | My description          | $11.35  | 01/01/2015 |
| 5  | Your description.       | $20     | 03/03/2013 |

，然后有两个可能的提取结果文件：

extract1.csv

| id | description             | date       |
|----|-------------------------|------------|
| 1  | Some description.       | 12/12/2012 |
| 2  | Some other description. | 10/10/2015 |
| 3  | Other description.      | 11/11/2014 |
| 4  | 122333222233332221      | 11/11/2014 |
| 5  | Your description.       | 03/03/2013 |

extract2.csv

| id | description             | amount  | date       |
|----|-------------------------|---------|------------|
| 1  | Some description.       | $150.54 | 12/12/2012 |
| 2  | Some other description. | $200    | 10/10/2015 |
| -  | ----------------------- | -----   | ---------- |
| 5  | Your description.       | $20     | 03/03/2013 |

extract3.csv

| Garbage  | More Garbage       |
| Garbage  | More Garbage       | 

我想让我的程序报告提取 1 缺少一列，并且列 2 中的值没有正确匹配。

对于第二种情况，我缺少条目并且某些行都不匹配。

在最后一种情况下，生成的 csv 都搞砸了，但我仍然希望程序能够检测到一些有意义的异常。

有没有人在 python 中有一些快速而聪明的方法来进行这种比较？

我有我可以在此处发布的常规、较长的逐行和逐列迭代方式，但我认为可能有一种更快、更优雅的 Pythonic 方式来执行此操作。

非常感谢任何帮助。

Answer 1

免责声明：我的方法使用pandas 库。

首先，数据设置。

gold_std.csv

id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
4,My description,$11.35,01/01/2015
5,Your description.,$20,03/03/2013

extract1.csv

id,description,date
1,Some description.,12/12/2012
2,Some other description.,10/10/2015
3,Other description.,11/11/2014
4,122333222233332221,11/11/2014
5,Your description.,03/03/2013

extract2.csv

id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
5,Your description.,$20,03/03/2013

第二，代码。

import pandas as pd

def compare_extract(extract_name, reference='gold_std.csv'):

    gold = pd.read_csv(reference)
    ext = pd.read_csv(extract_name)

    gc = set(gold.columns)
    header = ext.columns
    extc = set(header)

    if gc != extc:
        missing = ", ".join(list(gc - extc))
        print "Extract has the following missing columns: {}".format(missing)
    else:
        print "Extract has the same column as standard. Checking for abberant rows..."
        gold_list = gold.values.tolist()
        ext_list = ext.values.tolist()
        # Somewhat non-pandaic approach because possible no same IDs so we're relying
        # on set operations instead. A bit hackish, actually.
        diff = list(set(map(tuple, gold_list)) - set(map(tuple, ext_list)))
        df = pd.DataFrame(diff, columns=header)
        print "The following rows are not in the extract: "
        print df

第三，试运行。

e1 = 'extract1.csv'
compare_extract(e1)
# Extract has the following missing columns: amount

e2 = 'extract2.csv'
compare_extract(e2)
# Extract has the same column as standard. Checking for abberant rows...
# The following rows are not in the extract: 
#    id     description  amount        date
# 0   4  My description  $11.35  01/01/2015

最后，最后的摘录有点随意。我认为你最好写一个非pandas 算法。