如何计算数据框数组中元素的出现次数？答案

【问题标题】：How can I count occurrences of element in dataframe array?如何计算数据框数组中元素的出现次数？
【发布时间】：2021-10-16 07:49:21
【问题描述】：

我有一个如下所示的数据框：

df = spark.sql("""

SELECT list

FROM categories

""")

df.show()

list
1,1,1,2,2,apple
apple,orange,1,2

我想要这样的结果

list	frequency_count
1	4
2	3
apple	2
orange	1

这是我尝试过的。

count_df = df.withColumn('count', F.size(F.split('list', ',')))
count_df.show(truncate=False)

df.createOrReplaceTempView('tmp')
freq_sql = """
    select list,count(*) count from
        (select explode(flatten(collect_list(split(list, ',')))) list
        from tmp)
    group by list
"""
freq_df = spark.sql(freq_sql)
freq_df.show(truncate=False)

我收到了这个错误

AnalysisException: cannot resolve 'split(df.`list`, ',', -1)' due to
data type mismatch: argument 1 requires string type, however,
'df.`list`' is of array<string> type.;

【问题讨论】：

为什么不只是SELECT col, COUNT(*) FROM categories c LATERAL VIEW EXPLODE(list) l GROUP BY col ORDER BY col DESC。如果它已经是一个数组，不确定是否需要拆分它。

标签： apache-spark apache-spark-sql jupyter-notebook

【解决方案1】：

您当前正在尝试 flatten 单个 list 类型值，但是数组的 flatten 函数需要数组数组：

flatten(arrayOfArrays) - 将数组数组转换为单个数组。

因此您面临的结果错误

您需要先explode 列表，然后再尝试拆分它，最后分解它以将元素转换为行，最后groupBy 以获得所需的计数

数据准备

sparkDF = sql.createDataFrame(
        [
          (["1,1,1,2,2,apple"],),
          (["apple,orange,1,1"],),
        ],
        ("list",)
    )

爆炸

sparkDF.createOrReplaceTempView("dataset")

sql.sql("""
        SELECT
            explode(list) as exploded
            ,list
        FROM dataset
""").printSchema()

root
 |-- exploded: string (nullable = true)
 |-- list: array (nullable = true)
 |    |-- element: string (containsNull = true)


+----------------+------------------+
|        exploded|              list|
+----------------+------------------+
| 1,1,1,2,2,apple| [1,1,1,2,2,apple]|
|apple,orange,1,1|[apple,orange,1,1]|
+----------------+------------------+

分组方式

sql.sql("""
        SELECT
            exploded
            ,count(*) as count
        FROM (
            SELECT
                EXPLODE(SPLIT(list,",")) as exploded
            FROM (
                SELECT
                    EXPLODE(list) as list
                FROM dataset
            )
        )
        GROUP BY 1
""").show()

+--------+-----+
|exploded|count|
+--------+-----+
|  orange|    1|
|   apple|    2|
|       1|    5|
|       2|    2|
+--------+-----+

【讨论】：