【问题标题】:How do I map a numpy array and an indices array to a pandas dataframe?如何将 numpy 数组和索引数组映射到 pandas 数据帧?
【发布时间】:2021-11-14 15:15:23
【问题描述】:

我一直在关注这个tutorial,了解如何使用 scikit 找到一个点的最近邻居。

但是,在显示数据时,教程只提到“索引可以映射到有用的值,并且两个数组与其余数据合并”

但是没有关于如何执行此操作的实际解释。我对 Pandas 不是很精通,也不知道如何执行此合并,所以我最终得到了 2 个多维数组,我不知道如何将它们映射到原始数据以研究示例和尝试一下。

这是代码

import numpy as np
from sklearn.neighbors import BallTree, KDTree
import pandas as pd

# Column names for the example DataFrame.
column_names = ["STATION NAME", "LAT", "LON"]

# A list of locations that will be used to construct the binary
# tree.
locations_a = [['BEAUFORT', 32.4, -80.633],
       ['CONWAY HORRY COUNTY AIRPORT', 33.828, -79.122],
       ['HUSTON/EXECUTIVE', 29.8, -95.9],
       ['ELIZABETHTON MUNI', 36.371, -82.173],
       ['JACK BARSTOW AIRPORT', 43.663, -84.261],
       ['MARLBORO CO JETPORT H E AVENT', 34.622, -79.734],
       ['SUMMERVILLE AIRPORT', 33.063, -80.279]]

# A list of locations that will be used to construct the queries.
# for neighbors.
locations_b = [['BOOMVANG HELIPORT / OIL PLATFORM', 27.35, -94.633],
       ['LEE COUNTY AIRPORT', 36.654, -83.218],
       ['ELLINGTON', 35.507, -86.804],
       ['LAWRENCEVILLE BRUNSWICK MUNI', 36.773, -77.794],
       ['PUTNAM CO', 39.63, -86.814]]

# Converting the lists to DataFrames. We will build the tree with
# the first and execute the query on the second.

locations_a = pd.DataFrame(locations_a, columns = column_names)
locations_b = pd.DataFrame(locations_b, columns = column_names)

# Creates new columns converting coordinate degrees to radians.
for column in locations_a[["LAT", "LON"]]:
    rad = np.deg2rad(locations_a[column].values)
    locations_a[f'{column}_rad'] = rad
for column in locations_b[["LAT", "LON"]]:
    rad = np.deg2rad(locations_b[column].values)
    locations_b[f'{column}_rad'] = rad

# Takes the first group's latitude and longitude values to construct
# the ball tree.
ball = BallTree(locations_a[["LAT_rad", "LON_rad"]].values, metric='haversine')

# The amount of neighbors to return.
k = 1

# Executes a query with the second group. This will also return two
# arrays.
distances, indices = ball.query(locations_b[["LAT_rad", "LON_rad"]].values, k = k)
#converting to kilometers
distances = distances * 6.371

那么我该如何获取distancesindices 并将它们映射到我的数据框以直观地查看每个点的最近邻居?

【问题讨论】:

  • 注意:对于以公里为单位的距离,请从distances = distances * 6.371 中删除.

标签: python pandas dataframe numpy scikit-learn


【解决方案1】:

indices 中的每个整数索引都引用locations_a 的索引值(行号)。您可以使用locations_a.loc[] 将这些索引转换为它们对应的站名作为 numpy 数组:

nearest_station_names = locations_a.loc[indices.flatten()]['STATION NAME'].to_numpy()

(为什么indices.flatten() 而不仅仅是indicesball.querydistancesindices 作为二维numpy 数组返回,其中第二维(列数)为1。对于indices要在df.loc[] 中工作,您需要将其“展平”为一维数组,其唯一维度是行数。)

接下来,将名称作为新列插入locations_b

locations_b['nearest_stn'] = nearest_station_names

然后插入distances作为另一个新列(在这种情况下不需要.flatten):

locations_b['nearest_stn_dist'] = distances

# Print without radian columns for brevity
print(locations_b.drop(columns=['LAT_rad', 'LON_rad']))

                       STATION NAME     LAT     LON                    nearest_stn  nearest_stn_km
0  BOOMVANG HELIPORT / OIL PLATFORM  27.350 -94.633               HUSTON/EXECUTIVE      299.198339
1                LEE COUNTY AIRPORT  36.654 -83.218              ELIZABETHTON MUNI       98.550423
2                         ELLINGTON  35.507 -86.804              ELIZABETHTON MUNI      427.798176
3      LAWRENCEVILLE BRUNSWICK MUNI  36.773 -77.794  MARLBORO CO JETPORT H E AVENT      296.458070
4                         PUTNAM CO  39.630 -86.814           JACK BARSTOW AIRPORT      496.025005

【讨论】:

  • 谢谢,我现在明白了,使用locindices["STATION NAME]会在locations_a搜索索引值对应的每一行的指定列的值,但是我不明白locations_b['nearest_stn'] = nearest_station_names 如何准确地将行从location_a 分配到location_b。新列中的某些值是否存在与location_b 中的行不对应的风险?
  • 顺序将始终对齐,因为 ball.query 接受 locations_b 坐标的 numpy 数组,其中行按 DataFrame 中的出现顺序排序。然后,根据文档 (scikit-learn.org/stable/modules/generated/…),返回的 indices 数组的每个条目“给出对应点的邻居的索引列表。”
【解决方案2】:

根据BallTree() 的文档,indices 是一个形状为 (len(X), k) 的二维数组,其中X 是提供给查询中树的数组(这里是@987654324 @)。

因此,当您使用 locations_bk=1 进行查询时,您会收到一个二维数组 (5, 1),它表示 locations_b 的每个站点的最近邻居,作为原始 @987654328 的索引@ 用于适合BallTree,在本例中为locations_a

这意味着您现在拥有locations_a 的索引,它们代表locations_b 中每个站点的最近邻居。

对于k=1,您可以通过将locations_a 与您的查询产生的索引进行索引来获取有关最近车站的信息,如下所示:

neighbors = pd.DataFrame({'distance': distances.flatten(), 'neighbor_idx': indices.flatten()})

>>> neighbors

   distance  neighbor_idx
0  0.299198             2
1  0.098550             3
2  0.427798             3
3  0.296458             5
4  0.496025             4

在下面的代码中,我们joinlocations_bneighbors,默认加入索引,然后合并locations_a的一列,使用我们的neighbor_idx字段匹配locations_a的索引.

locations_b\
    .join(neighbors)\
    .merge(locations_a[['STATION NAME']], left_on='neighbor_idx', right_index=True, suffixes=("", "_neighbor"))\
    .drop('neighbor_idx', axis=1)
                       STATION NAME     LAT     LON   LAT_rad   LON_rad  distance          STATION NAME_neighbor
0  BOOMVANG HELIPORT / OIL PLATFORM  27.350 -94.633  0.477348 -1.651657  0.299198               HUSTON/EXECUTIVE
1                LEE COUNTY AIRPORT  36.654 -83.218  0.639733 -1.452428  0.098550              ELIZABETHTON MUNI
2                         ELLINGTON  35.507 -86.804  0.619714 -1.515016  0.427798              ELIZABETHTON MUNI
3      LAWRENCEVILLE BRUNSWICK MUNI  36.773 -77.794  0.641810 -1.357761  0.296458  MARLBORO CO JETPORT H E AVENT
4                         PUTNAM CO  39.630 -86.814  0.691674 -1.515190  0.496025           JACK BARSTOW AIRPORT

您当然可以选择合并来自locations_a 的其他列。

在多个最近邻的一般情况下,上述方法不会完全正确。以下是解决此问题的方法:

k=3
distances, indices = ball.query(locations_b[["LAT_rad", "LON_rad"]].values, k = k)
#converting to kilometers
distances = distances * 6.371

dists = pd.DataFrame(distances).stack()
rel = pd.DataFrame(indices).stack()
neighbor_df = pd.merge(dists.rename('distance'), rel.rename('neighbor_idx'), right_index=True, left_index=True)
neighbor_df = neighbor_df.reset_index(level=1)
neighbor_df.columns = ['neighbor_number', 'distance', 'neighbor_idx']

>>> neighbor_df

   neighbor_number  distance  neighbor_idx
0                0  0.299198             2
0                1  1.460424             0
0                2  1.516741             6
1                0  0.098550             3
1                1  0.387486             5
1                2  0.480926             6
2                0  0.427798             3
2                1  0.650799             5
2                2  0.658009             6
3                0  0.296458             5
3                1  0.348930             1
3                2  0.393563             3
4                0  0.496025             4
4                1  0.544545             3
4                2  0.838587             5

locations_b\
    .join(neighbor_df)\
    .merge(locations_a[['STATION NAME']], left_on='neighbor_idx', right_index=True, suffixes=("", "_neighbor"))\
    .drop('neighbor_idx', axis=1)

结果:


                       STATION NAME     LAT     LON   LAT_rad   LON_rad  neighbor_number  distance          STATION NAME_neighbor
0  BOOMVANG HELIPORT / OIL PLATFORM  27.350 -94.633  0.477348 -1.651657                0  0.299198               HUSTON/EXECUTIVE
0  BOOMVANG HELIPORT / OIL PLATFORM  27.350 -94.633  0.477348 -1.651657                1  1.460424                       BEAUFORT
0  BOOMVANG HELIPORT / OIL PLATFORM  27.350 -94.633  0.477348 -1.651657                2  1.516741            SUMMERVILLE AIRPORT
1                LEE COUNTY AIRPORT  36.654 -83.218  0.639733 -1.452428                2  0.480926            SUMMERVILLE AIRPORT
2                         ELLINGTON  35.507 -86.804  0.619714 -1.515016                2  0.658009            SUMMERVILLE AIRPORT
1                LEE COUNTY AIRPORT  36.654 -83.218  0.639733 -1.452428                0  0.098550              ELIZABETHTON MUNI
2                         ELLINGTON  35.507 -86.804  0.619714 -1.515016                0  0.427798              ELIZABETHTON MUNI
3      LAWRENCEVILLE BRUNSWICK MUNI  36.773 -77.794  0.641810 -1.357761                2  0.393563              ELIZABETHTON MUNI
4                         PUTNAM CO  39.630 -86.814  0.691674 -1.515190                1  0.544545              ELIZABETHTON MUNI
1                LEE COUNTY AIRPORT  36.654 -83.218  0.639733 -1.452428                1  0.387486  MARLBORO CO JETPORT H E AVENT
2                         ELLINGTON  35.507 -86.804  0.619714 -1.515016                1  0.650799  MARLBORO CO JETPORT H E AVENT
3      LAWRENCEVILLE BRUNSWICK MUNI  36.773 -77.794  0.641810 -1.357761                0  0.296458  MARLBORO CO JETPORT H E AVENT
4                         PUTNAM CO  39.630 -86.814  0.691674 -1.515190                2  0.838587  MARLBORO CO JETPORT H E AVENT
3      LAWRENCEVILLE BRUNSWICK MUNI  36.773 -77.794  0.641810 -1.357761                1  0.348930    CONWAY HORRY COUNTY AIRPORT
4                         PUTNAM CO  39.630 -86.814  0.691674 -1.515190                0  0.496025           JACK BARSTOW AIRPORT

酷项目!

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2014-08-25
    • 2019-09-23
    • 2017-07-07
    • 2021-01-29
    • 1970-01-01
    • 2017-04-20
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多