Pyspark 错误：TypeError：不能将类型 <type 'NoneType'> 视为向量 [重复]

Question

我有以下代码：

from pyspark.sql.types import *
from pyspark.ml.linalg import Vectors, VectorUDT
import pyspark.sql.functions as F

dot_udf = F.udf(lambda x,y: float(x.dot(y)), DoubleType())
l = [(Vectors.dense([1, 2, 3, 4 ,5]), Vectors.dense([5, 4, 3, 2, 1]),),
     (Vectors.dense([0, 4, 8, 2, 1]), None,),
     (None, Vectors.dense([5, 0, 3, 9, 1]),),
     ]

def finish(row):
    new_row = []
    new_row.append(None if row['my_row_1'] == None else Vectors.dense(row['my_row_1']))
    new_row.append(None if row['my_row_2'] == None else Vectors.dense(row['my_row_2']))
    return new_row


with (SparkSession
      .builder
      .appName('test_mtassoni')
      .getOrCreate()) as spark:
    schema = StructType([StructField('my_row_1', VectorUDT(), True),
                         StructField('my_row_2', VectorUDT(), True)])

    df = spark.createDataFrame(l, schema)
    rdd = df.rdd
    rdd = rdd.map(finish)

    out_schema = StructType([StructField('my_row_1', VectorUDT(), True),
                             StructField('my_row_2', VectorUDT(), True)])
    fdf = spark.createDataFrame(rdd, schema=out_schema)

    fdf = fdf.withColumn('row_sim', F.when(((F.col('my_row_1').isNull()) |
                                            (F.col('my_row_2').isNull())),
                                       np.nan).otherwise(dot_udf(fdf.my_row_1, fdf.my_row_2))
                     )
    fdf.show()

在最后一个命令上出现以下 TypeError 失败：

TypeError: Cannot treat type <type 'NoneType'> as a vector

有人知道如何解决吗？非常感谢您提前。

Answer 1

您的方法的问题是您在otherwise 部分调用了udf 函数，但事实是每一行都被传递给udf 函数。所以问题在于您将 column-wise 函数 (when/otherwise) 与 row-wise 函数 (udf)

解决方案是将when/otherwise 部分移动到udf 函数中

def dotProduct(x, y):
    if(x == None or y == None):
        return np.nan
    else:
        return float(x.dot(y))

dot_udf = F.udf(lambda x,y: dotProduct(x, y), DoubleType())

然后调用udf函数独立为

fdf = fdf.withColumn('row_sim', dot_udf(fdf.my_row_1, fdf.my_row_2))
fdf.show()

这应该会给你正确的结果而没有错误

+--------------------+--------------------+-------+
|            my_row_1|            my_row_2|row_sim|
+--------------------+--------------------+-------+
|[1.0,2.0,3.0,4.0,...|[5.0,4.0,3.0,2.0,...|   35.0|
|[0.0,4.0,8.0,2.0,...|                null|    NaN|
|                null|[5.0,0.0,3.0,9.0,...|    NaN|
+--------------------+--------------------+-------+

希望回答对你有帮助