在 CSV 中查找每列的“最强”类型

Question

我需要按列扫描 CSV 并找到最强的数据类型，然后将其应用于整个列。

例如，如果我有一个看起来像这样的 CSV（是的，我没有逗号...）：

    + C1 + C2 + C3 + C4
R1  | i  | s  | i  | f
R2  | i  | f  | i  | i
R3  | i  | i  | s  | f

# i = int
# f = float
# s = str

C1 的“最强”类型是 i，C2 是 s，C3 是 s，C4 是 f。

因此“强度”的顺序是str > float > int。

为什么？因为我正在写入这些值的文件类型明确要求为字段（它的列）指定的数据类型与该数据类型匹配（即，如果该字段设置为FLOAT，我不能在其中放置str列，否则文件无效）。

为此，我正在执行以下操作：

1. 对于每个文件，逐行读取文件并检查每一列；存储“最强”类型
2. 创建一个包含新类型转换行的新容器

第 2 项使用字典和列表理解非常简单：

types = {header: None for header in r.fieldnames}
# read file and store "strongest" found in 'types[header]' per column
# ...
typed = [[types[header](row[header]) for header in types] for row in rows]
# note: types[header] value is a function alias (i.e. int vs int())

第 1 项是大部分繁重工作发生的地方：

for row in r: # r is a csv.DictReader
    rows.append(row) # list of OrderedDicts since r is a generator
    # problematic because I have to keep checking just to append...
    if all(types[header] is str for header in types):
        continue # all 'str' so stop checking

    for header in types:
        if types[header] is str:
            continue # whole column can be bypassed from now on

        # function just type casts 'int' or 'float' on string by ValueError
        t = self.find_type(row[header])
        if (types[header] is int) and (t is float):
            types[header] = t # float > int since all int's can be represented as float
        elif (types[header] is float) and (t is int):
            pass # int < float so do nothing
        else:
            types[header] = t # if 'str' will be caught later by first if

执行此操作的最坏情况是 CSV 中的行数，因为最后一行可能包含有效的 str 类型测试。

有没有更有效的方法来做到这一点，也许是pandas（目前使用不多）？

解决方案：

from numpy import issubdtype
from numpy import integer
from numpy import floating
from pandas import read_csv
from shapefile import Writer # PyShp library

df = read_csv('/some/file', low_memory = False)

rows = df.values.tolist() # fastest access over df.iterrows()

w = Writer(5, True)

# This is the core of the question
# I can access df[col].dtype but I didn't want to use str == str
# If this can be done better than subtype check let me know
for col in df:
    if issubdtype(df[col], integer): 
        w.field(col, 'N', 20, 0)
    elif issubdtype(df[col][0], floating):
        w.field(col, 'F', 20, 10)
    else:
        w.field(col, 'C', 40, 0)

# Alternatively (1):
# from numpy import int64
# from numpy import float64
# for col in df:
#     if df[col].dtype.type is int64: 
#         w.field(col, 'N', 20, 0)
#     elif df[col].dtype.type is float64:
#         w.field(col, 'F', 20, 10)
#     else:
#         w.field(col, 'C', 40, 0)

# Alternatively (2):
# Don't import numpy directly in namespace
# for col in df:
#     if df[col].dtype == 'int64': 
#         w.field(col, 'N', 20, 0)
#     elif df[col].dtype == 'float64':
#         w.field(col, 'F', 20, 10)
#     else:
#         w.field(col, 'C', 40, 0)


lon = df.columns.get_loc('LON')
lat = df.columns.get_loc('LAT')

for row in rows:
    w.point(row[lon], row[lat])
    w.record(*row)

w.save('/some/outfile')

Answer 1

一个示例数据框：

In [11]: df
Out[11]:
    C1  C2 C3    C4
R1   1   a  6   8.0
R2   2  4.  7   9.0
R3   3   5  b  10.0

我不会尝试对任何短路评估保持聪明。我只需要每个条目的类型：

In [12]: df_types = df.applymap(type)

In [13]: df_types
Out[13]:
               C1             C2             C3               C4
R1  <class 'int'>  <class 'str'>  <class 'str'>  <class 'float'>
R2  <class 'int'>  <class 'str'>  <class 'str'>  <class 'float'>
R3  <class 'int'>  <class 'str'>  <class 'str'>  <class 'float'>

如果您枚举这些类型，您可以使用max：

In [14]: d = {ch: i for i, ch in enumerate([int, float, str])}

In [15]: d_inv = {i: ch for i, ch in enumerate([int, float, str])}

In [16]: df_types.applymap(d.get)
Out[16]:
    C1  C2  C3  C4
R1   0   2   2   1
R2   0   2   2   1
R3   0   2   2   1

In [17]: df_types.applymap(d.get).max()
Out[17]:
C1    0
C2    2
C3    2
C4    1
dtype: int64

In [18]: df_types.applymap(d.get).max().apply(d_inv.get)
Out[18]:
C1      <class 'int'>
C2      <class 'str'>
C3      <class 'str'>
C4    <class 'float'>
dtype: object

现在，您可以遍历每一列并在 df 中更新它（最大）：

In [21]: for col, typ in df_types.applymap(d.get).max().apply(d_inv.get).iteritems():
             df[col] = df[col].astype(typ)


In [22]: df
Out[22]:
    C1  C2 C3    C4
R1   1   a  6   8.0
R2   2  4.  7   9.0
R3   3   5  b  10.0

In [23]: df.dtypes
Out[23]:
C1      int64
C2     object
C3     object
C4    float64
dtype: object

如果您通过按类型分组并更新成批的列（例如一次所有字符串列）有许多列，这可能会稍微更有效。