使用 genfromtxt() 和转换器导入 csv答案

【问题标题】：Import csv using genfromtxt() and converters使用 genfromtxt() 和转换器导入 csv
【发布时间】：2021-03-12 14:43:59
【问题描述】：

import numpy as np
import csv
filename = "a.csv"

def convert(s): 
    s = s.strip().replace(',', '.')
    return str(s)

salary_data = np.genfromtxt(filename,
                     delimiter= ',',
                     dtype=[('year','i8'),('university','U50'),('school','U250'), 
                     ('degree','U250'),('employement_rate_overall','f8'), 
                     ('basic_monthly_mean','f8'),('gross_monthly_mean','i8'), 
                     ('gross_monthly_median','i8'),('gross_mthly_25_percentile','i8'), 
                     ('gross_mthly_75_percentile','i8')], 
                     encoding= None, #avoid having the deprecated warning
                     skip_header=1,
                     missing_values=['na','-'],filling_values=[0],
                     converters={2: convert} ,
                     comments=None)
print(salary_data)

我试图加载 csv 数据，但数据很脏，因为它在某些值字段中包含引号/逗号，并导致我出错。

      Some errors were detected!
      Line #5 (got 13 columns instead of 12)

我试图通过使用转换器来清理逗号。但是，代码似乎不起作用。我试过了

      converters={2: lambda s: str(s.replace(',', '.'))}

这也不适用于我的情况。我希望知道我的错误是什么，并感谢您的帮助！感谢那些发现我的错误的人！即使我试图替换引号，代码也不起作用。下面的文本是我正在加载的 csv 文件。

      year,university,school,degree,employment_rate_overall,employment_rate_ft_perm,basic_monthly_mean,basic_monthly_median,gross_monthly_mean,gross_monthly_median,gross_mthly_25_percentile,gross_mthly_75_percentile
     2013,Nanyang Technological University,College of Business (Nanyang Business School),Accountancy and Business,97.4,96.1,3701,3200,3727,3350,2900,4000
     2013,Nanyang Technological University,College of Business (Nanyang Business 
     School),Accountancy (3-yr direct Honours Programme),97.1,95.7,2850,2700,2938,2700,2700,2900
     2013,Nanyang Technological University,College of Business (Nanyang Business 
     School),Business (3-yr direct Honours Programme),90.9,85.7,3053,3000,3214,3000,2700,3500
     2013,Nanyang Technological University,"College of Humanities, Arts & Social 
     Sciences",Economics,89.9,83.5,3085,3000,3148,3000,2800,3545
     2013,Nanyang Technological University,College of Sciences,Biomedical Sciences 
     **,na,na,na,na,na,na,na,na
     2013,Nanyang Technological University,College of Sciences,Biomedical Sciences 
     (Traditional Chinese Medicine) #,90.7,88.4,2840,2800,2883,2807,2700,3000
     2013,Nanyang Technological University,College of Sciences,Mathematics & Economics 
     **,na,na,na,na,na,na,na,na
     2014,Nanyang Technological University,"College of Humanities, Arts & Social 
     Sciences","Art, Design & Media",80,68,2761,2600,2791,2700,2300,3000

【问题讨论】：

在您的原始 .csv 文件中是否存在或多或少的随机换行符，将实际单个项目分成两行？例如2013,Nanyang Technological University,"College of Humanities,结尾的情况似乎就是这样
是肯定的.. 有没有办法在 gen from txt 函数中解决它？

标签： python converters genfromtxt

【解决方案1】：

我将您的文件作为 .csv 导入，正如 @fischmalte 指出的那样，有新行，例如 Nanyang Business School。

但这不会导致您的错误。
其实错误Line #5 (got 13 columns instead of 12)，是由"College of Humanities, Arts & Social Sciences"的"引起的

因此，csv 阅读器会再生成一列。
删除它们，您的错误就会消失。

另外，如果你使用 pandas，" 将被自动处理：

import pandas as pd
df = pd.DataFrame("my_file.csv")

（虽然它不会处理线路断路器）

【讨论】：