遍历 ndarray 类型的 pandas 列答案

【问题标题】：iterate over pandas columns of type ndarray遍历 ndarray 类型的 pandas 列
【发布时间】：2021-04-30 11:54:46
【问题描述】：

我有一个包含两列的数据框，第一列是 id，另一列是数组。此列是一个包含 NaN 值的数组 (ndarray)。（我已经完成了转换它的第一步，因为它是以字节为单位的。在进行此更改时，列的类型可能是 str。）

terrain['ndvi_matrix'] = np.array(terrain.ndvi_matrix.str.decode('utf-8'))
terrain

terrain_id  ndvi_matrix
0   1   [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
1   2   [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
2   3   [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
3   4   [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
4   5   [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
5   6   [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
6   7   [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
7   23  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
8   27  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
9   28  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
10  35  [[NaN, NaN, NaN, NaN, NaN, NaN], [0.3127734033...
11  36  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
12  42  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
13  50  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
14  51  [[NaN, 0.18129175946547885, 0.1526586620926243...
15  52  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
16  55  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
17  56  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
18  57  [[NaN, NaN, NaN, NaN, NaN, NaN], [NaN, 0.38725...
19  58  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
20  59  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
21  61  [[NaN, NaN, NaN, NaN, NaN, NaN], [NaN, NaN, Na...
22  62  [[NaN, NaN, NaN, NaN, NaN, NaN], [0.0791249233...
23  63  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
24  64  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN,...
25  67  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
26  68  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
27  73  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
28  74  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
29  77  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
30  79  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...
31  80  [[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694...

terrain.dtypes
terrain_id      int64
ndvi_matrix    object
dtype: object

我想要的是迭代这些行的值并放置条件。例如，对于大于 0.5 的值，将值更新 0.99。我也遇到了 NaN 的问题，因为当我使用 for 和 iterrows 进行迭代时，它告诉我该列的类型是 str。

我已经尝试过了，但它没有更新数据帧（在这种情况下，我使用 jsonloads 进行编码）

for index, row in terrain.iterrows():
  terrain_id = row['terrain_id']
  ndvi_matrix = np.array(json.loads(row['ndvi_matrix']))
  for col in ndvi_matrix:
    for pixel in col:
      if pixel > 0.5:
        pixel = 0.99
      elif 0.2 < pixel < 0.5:
        Fc = ((pixel - 0.2) ** 2)/0.09 
        pixel = Fc
      elif pixel < 0.2:
        pixel = 0.944
      else:
        pixel = 'nan'
      
      print(pixel)

示例：

array(['[[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694915254237, 0.3121476166068683, 0.43469446857804805, 0.35864022662889516, NaN, NaN], [NaN, NaN, NaN, 0.39893805309734515, 0.5044690740078656, 0.5433070866141733, 0.5176265270506109, 0.4300861497680583, 0.31038343129023227, 0.41292379020573744, 0.4365527488855869, NaN, NaN], [NaN, NaN, NaN, 0.5380281690140845, 0.5721357850070722, 0.5886981402002861, 0.5908611599297012, 0.6074124914207275, 0.6179540709812108, 0.6184530003468609, 0.6147846727704307, 0.5858347386172007, NaN], [NaN, NaN, NaN, 0.605643994211288, 0.613724796604174, 0.6004250797024442, 0.603299856527977, 0.609429978888107, 0.6084546084546084, 0.626489138051857, 0.6185567010309279, 0.5827633378932968, NaN], [NaN, NaN, 0.6287215411558669, 0.6118980169971672, 0.6051532033426184, 0.6032303370786517, 0.6175839885632595, 0.6167146974063401, 0.6188786373314408, 0.628808864265928, 0.6193103448275862, 0.5957671957671957, NaN], [NaN, NaN, 0.6093189964157706, 0.5889921372408864, 0.5956552207428171, 0.608649415101028, 0.6209705986539142, 0.6205533596837944, 0.6292857142857143, 0.6296036478428622, 0.6120659417748159, 0.5862311204776958, NaN], [NaN, NaN, 0.59375, 0.5653429602888087, 0.589817138759412, 0.5995694294940797, 0.607563325008919, 0.6096333572969087, 0.6217345872518286, 0.6243441762854145, 0.6021390374331551, 0.6045118082481494, NaN], [NaN, NaN, 0.5848920863309353, 0.5734011627906976, 0.6040100250626567, 0.6092691622103387, 0.6002865329512894, 0.582879941965905, 0.5781518378564068, 0.6072676450034941, 0.5981375358166189, 0.6016771488469602, NaN], [NaN, 0.5665236051502146, 0.5756026296566837, 0.5879140880961048, 0.615983026874116, 0.6137339055793991, 0.5862068965517241, 0.5815808556925308, 0.585934696806602, 0.5968034871049764, 0.5933789132781576, 0.5913073957237995, NaN], [NaN, 0.5614353026458861, 0.582338038364097, 0.5881929445644348, 0.6124737210932025, 0.606952820148989, 0.573746844572665, 0.6049733570159858, 0.6176678445229682, 0.6090491339696006, 0.6205092431112661, 0.6100981767180925, NaN], [NaN, 0.5616045845272206, 0.5915744377008212, 0.5772870662460567, 0.5922535211267606, 0.5998591053187742, 0.5921513665031535, 0.6301703163017032, 0.6083421703782255, 0.6028169014084507, 0.6353629170966633, 0.6252602359472589, NaN], [NaN, 0.5752895752895753, 0.6055944055944056, 0.5848400556328234, 0.587513153279551, 0.6028169014084507, 0.6085902320748181, 0.6206185567010309, 0.6098591549295774, 0.5980528511821975, 0.6147994467496543, 0.6101283385362469, NaN], [NaN, 0.6063756063756064, 0.6014735432016075, 0.5934178950977032, 0.6148300720906282, 0.5845942228335625, 0.5820486467968482, 0.6119917298414886, 0.5669050051072523, 0.5611390284757118, 0.5912897822445561, 0.564563582870219, NaN], [0.4517675254643499, 0.4760514018691589, 0.4551226551226551, 0.414577931431864, 0.4381895332390382, 0.42126436781609194, 0.3607280750137893, 0.3523246439129266, 0.3677453864669698, 0.3598930481283422, 0.34667359667359665, 0.3580502981591911, NaN], [NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]]'],
      dtype=object)

【问题讨论】：

ndvi_matrix 是字符串还是numpy数组？
我转换为 np.array 但我不知道它是如何获得 NaN 值的，我猜是 str 格式
您的输入数据不可用。复制/粘贴一个完整的原始字符串。我们无法猜测每一行的形状。
我已经用一个例子编辑了这个问题
解码前能不能有原来的terrain['ndvi_matrix']？

标签： python pandas dataframe

【解决方案1】：

这样的事情有帮助吗？这将使用函数中定义的任何应用逻辑创建一个新列。

import numpy as np
import pandas as pd

d = {'arr': [np.array([[np.nan, .55, .02 , .88]]), np.array([[np.nan, .55, .02 , .88]])]}

terrain= pd.DataFrame(d, columns=['arr'])

def arr_logic(arr):
    new_arr = []
    for pixel in arr[0]:
        if pixel > 0.5:
            new_arr.append(0.99)
        elif 0.2 < pixel < 0.5:
            Fc = ((pixel - 0.2) ** 2)/0.09
            new_arr.append(Fc)
        elif pixel < 0.2:
            new_arr.append(0.944)
        else:
            new_arr.append(pixel)

    return [new_arr]

terrain['new_arr'] = terrain['arr'].apply(lambda x: arr_logic(x))
print(terrain.dtypes)
print(terrain)


arr        object
new_ary    object
dtype: object
                         arr                     new_arr
0  [[nan, 0.55, 0.02, 0.88]]  [[nan, 0.99, 0.944, 0.99]]
1  [[nan, 0.55, 0.02, 0.88]]  [[nan, 0.99, 0.944, 0.99]]
[Finished in 0.55s]

这是您可以对您提供的示例数组进行的一些处理。在 df 中迭代之前，最好使用正确的格式提取数组。


import numpy as np
import pandas as pd
import re

d = {'arr': [b'[[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694915254237, 0.3121476166068683, 0.43469446857804805, 0.35864022662889516]]',
 b'[[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.3601694915254237, 0.3121476166068683, 0.43469446857804805, 0.35864022662889516]]']}

terrain= pd.DataFrame(d, columns=['arr'])

def arr_logic(arr):
    new_arr = []
    arr = re.sub(r"\[|\]", '', arr)
    arr = [np.nan if i =='NaN' else float(i) for i in arr.split(",")]

    for pixel in arr:
        if pixel > 0.5:
            new_arr.append(0.99)
        elif 0.2 < pixel < 0.5:
            Fc = ((pixel - 0.2) ** 2)/0.09
            new_arr.append(Fc)
        elif pixel < 0.2:
            new_arr.append(0.944)
        else:
            new_arr.append(pixel)

    return [new_arr]

terrain['arr'] = terrain['arr'].str.decode('utf-8')
terrain['new_ary'] = terrain['arr'].apply(lambda x: arr_logic(x))

print(terrain.dtypes)
print(terrain)

【讨论】：

这很好，但它会产生一个问题，我已经评论了TypeError: '>' not supported between instances of 'str' and 'float'@KJDII 这是由于 NaN 值。是否将 NaN 值视为 str
在创建阵列之前，您是否有地形样本['ndvi_matrix']？听起来您可能需要预处理该信息以获取 np.nan 和 floats
我只能得到这个，我不能完整得到它b'[[NaN, NaN, NaN, NaN, NaN, NaN, NaN, 0.36016..
更新了b'[[NaN, ... 例子的句柄。
谢谢，这段代码帮助了我。我不知道可以在 ndarrays 中使用 lambda 函数

【解决方案2】：

输入数据：

>>> terrain
   terrain_id  ndvi_matrix
0           5  [[0.8217155552979787, 0.3099672115904418, 0.6266277523163207], [0.11725795982601384, 0.7234050380206947, 0.23432585428525776], [0.14658855068952603, 0.9934660058726204, 0.5843251036416421]]
1           9  [[0.20367346628943583, NaN, 0.8183039429570168], [NaN, NaN, NaN], [0.3480401851147674, NaN, NaN], [0.3897797899361942, 0.9695928756461897, 0.006497296226907423], [0.7678232526089657, 0.017039908371399903, 0.07803224625341965]]
2          10  [[0.8110310466645949, 0.5694625468868105], [0.330177274386086, 0.8385546531259933], [0.8823223193531003, 0.6800801476410133], [NaN, NaN]]
3          18  [[0.5399031981387157, NaN, 0.9930978186475411, 0.3830175767189744, 0.7685109432532721], [0.022656103788886273, 0.3860560747385754, 0.9699605735947163, 0.036483440273193946, 0.13789939109679117]]
4          19  [[NaN, 0.5062668350287397, 0.05922667405589466, NaN], [0.706118039096498, 0.13114569212422644, 0.9806901059099363, NaN], [0.9248269745262615, 0.6596793327690313, 0.43770577542377953, 0.11179684532285561]]
5          23  [[NaN, 0.6472524121862453, 0.29064151921529846, 0.49507164285810745], [0.6584281587251608, 0.8381562631806461, NaN, 0.4492074246342629]]
6          24  [[0.3711511153060173, NaN, 0.19832511498034666, 0.8604071320413275, 0.45380975705391313], [0.709501182520755, 0.590680803287601, 0.7832281301037608, 0.1150537088106176, 0.48021625141228275], [0.45883798079380966, NaN, 0.35899605566841897, NaN, 0.05813644920490424], [0.22258381790125448, NaN, 0.48509936372932805, 0.48675292417904825, 0.3110740604822009]]
7          26  [[NaN, 0.885255163613397], [0.6864001511014625, NaN], [0.5793096811692588, NaN], [NaN, 0.8727310784076551], [0.650092442108687, NaN]]
8          29  [[NaN, 0.8262184756213229, NaN, 0.3138323965010388, NaN], [0.644053291959643, 0.5339976526420492, NaN, 0.7486133307776289, NaN], [0.2573179543021944, NaN, 0.36980289291571045, NaN, NaN], [0.9866585953070125, NaN, 0.18845706290927178, 0.986040466809015, NaN]]
9          36  [[NaN, 0.3351322849176379, NaN, 0.6663735283696609, NaN], [0.6755852782079985, NaN, NaN, 0.8554058123116294, 0.9132271353603988], [0.7249084160159946, 0.691724896507485, NaN, 0.5223203096525957, NaN]]

我假设每行的ndvi_matrix 类型为str，因为您使用的是json.loads()。
我猜您将列表表示形式转换为np.array。

>>> terrain.loc[9, "ndvi_matrix"]
'[[NaN, 0.3351322849176379, NaN, 0.6663735283696609, NaN], [0.6755852782079985, NaN, NaN, 0.8554058123116294, 0.9132271353603988], [0.7249084160159946, 0.691724896507485, NaN, 0.5223203096525957, NaN]]'

>>> type(terrain.loc[9, "ndvi_matrix"])
str

你只需要另一个转换步骤：

terrain["ndvi_matrix"] = terrain["ndvi_matrix"].map(lambda a: np.array(eval(a.replace("NaN", "'nan'")), dtype=float))

>>> terrain.loc[9, "ndvi_matrix"]
array([[       nan, 0.33513228,        nan, 0.66637353,        nan],
       [0.67558528,        nan,        nan, 0.85540581, 0.91322714],
       [0.72490842, 0.6917249 ,        nan, 0.52232031,        nan]])

>>> type(terrain.loc[9, "ndvi_matrix"])
numpy.ndarray

现在您可以处理您的数据了：

def arr_logic(arr):
    new_arr = np.empty(arr.shape)
    new_arr[:] = np.nan

    # boolean indexing
    m1 = arr > 0.5
    m2 = (0.2 < arr) & (arr < 0.5)
    m3 = arr < 0.2

    # !!! When arr==0.2 or arr==0.5 then new_arr=nan !!!

    # apply your rules
    new_arr[m1] = 0.99
    new_arr[m2] = (arr[m2] - 0.2)**2 / 0.09
    new_arr[m3] = 0.944

    # all other values are already nan
    return new_arr

terrain["ndvi_matrix"] = terrain["ndvi_matrix"].apply(arr_logic)

输出数据：

>>> terrain
terrain_id     ndvi_matrix
0           5  [[0.99, 0.1343643069441889, 0.99], [0.944, 0.99, 0.013091825249030532], [0.944, 0.99, 0.99]]
1           9  [[0.00014993727310690436, nan, 0.99], [nan, nan, nan], [0.24350996009794001, nan, nan], [0.40018187409140005, 0.99, 0.944], [0.99, 0.944, 0.944]]
2          10  [[0.99, 0.99], [0.1882902529621147, 0.99], [0.99, 0.99], [nan, nan]]
3          18  [[0.99, nan, 0.99, 0.3721714820898409, 0.99], [0.944, 0.3846318105236261, 0.99, 0.944, 0.944]]
4          19  [[nan, 0.99, 0.944, nan], [0.99, 0.944, 0.99, nan], [0.99, 0.99, 0.6278226185535589, 0.944]]
5          23  [[nan, 0.99, 0.0912876111739702, 0.9674141602109168], [0.99, 0.99, nan, 0.6900482276982424]]
6          24  [[0.3254744918943736, nan, 0.944, 0.99, 0.7157710308418489], [0.99, 0.99, 0.99, 0.944, 0.872457195061685], [0.7444122255712953, nan, 0.2808860635346109, nan, 0.944], [0.005666987011078019, nan, 0.9031294133207521, 0.9136359947248333, 0.13708274346670685]]
7          26  [[nan, 0.99], [0.99, nan], [0.99, nan], [nan, 0.99], [0.99, nan]]
8          29  [[nan, 0.99, nan, 0.1439757165907745, nan], [0.99, 0.99, nan, 0.99, nan], [0.03650386539320493, nan, 0.32036691602826917, nan, nan], [0.99, nan, 0.944, 0.99, nan]]
9          36  [[nan, 0.2028970491895741, nan, 0.99, nan], [0.99, nan, nan, 0.99, 0.99], [0.99, 0.99, nan, 0.99, nan]]

【讨论】：

谢谢。但是我使用了来自@KJDII 的代码，因为转换对我有用，但是 map 函数不允许我在我的代码中使用它。很抱歉，我没有完全解释我的数据。无论如何， arr_logic 函数对我有用。谢谢