从 pyspark 中的数据框中提取数据

Question

我正在尝试在 pyspark 中创建一个脚本，该脚本将从表中获取最小和最大日期并将它们存储在 df 中，然后将这两个值拆分为 2 个变量，然后将这些变量作为时间范围放置在另一个查询中。我的问题是日期是这样的数据框

+--------+--------+
| maxDate| minDate|
+--------+--------+
|20210701|20210629|
+--------+--------+

我只想要 maxDate 和 minDate 的值。

我尝试了dates.iloc[0] 和var1 = dates['maxDate'].values[0]，但没有成功。

from pyspark.sql import SparkSession
from pyspark.sql.functions import when
from pyspark.sql import functions as F
from pyspark.sql.functions import lit
from pyspark.sql.functions import trim
from datetime import datetime


current_timestamp = datetime.strftime(datetime.now(), "%Y%m%d%H%M")

spark = SparkSession.builder.appName("testing") \
.config("hive.exec.dynamic.partition", "true") \
.config("hive.exec.dynamic.partition.mode", "nonstrict") \
.config("hive.exec.compress.output=false", "false") \
.config("spark.unsafe.sorter.spill.read.ahead.enabled", "false") \
.config("spark.debug.maxToStringFields", 1000)\
.enableHiveSupport() \
.getOrCreate()

spark.sql("set max_row_size = 6mb")
dates = spark.sql("SELECT MAX(date) as maxDate, MIN(date) as minDate FROM db.table")
 
#dates must be split here in two separated vars

result = spark.sql("select * from db.table_2 where date between {} and {}".format(var1,var2)

Answer 1

你可以像下面那样做

max_date = df.collect()[0][0]

min_date = df.collect()[0][1]