如何在foreachpartition中获取行属性值

Question

我在努力

def customFunction(rows):
    for row in rows:
        key = row.key #this value is boolean instead of actual value same with row["key"]
        val = row.value #this value is boolean instead of actual value same with row["val"]
        #do something with key value

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL Hive integration example") \
    .config("spark.sql.warehouse.dir", warehouse_location) \
    .enableHiveSupport() \
    .getOrCreate()


# spark is an existing SparkSession
spark.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive")
spark.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src")

# Queries are expressed in HiveQL
df = spark.sql("SELECT key, value FROM src")

# assumption that df row size is of billions
df.rdd.foreachPartition(customFunction)

我在自定义函数中的键、val 变量中获取布尔值。我们如何获取行属性实际值？

这是在 aws emr 5.29、python 2.7 上运行的，python 代码是通过 spark-submit 执行的

Answer 1

如果这有帮助，在 customFunction 内部，我试图将值 key 用于 dynamodb 并且当键为 NULL 时它正在存储一个布尔值。