不使用spark sql函数的每个部门的明智工资[关闭]答案

【问题标题】：Designation wise Salary of each department without using spark sql function [closed]不使用spark sql函数的每个部门的明智工资[关闭]
【发布时间】：2021-08-28 15:50:54
【问题描述】：

显示每个职位名称的部门明智总薪水，例如初级、女售货员、主管。 *不使用 spark.sql()

输入

+-----+------+--------+----+----+----+------+
|empno| ename|     job| mgr| sal|comm|deptno|
+-----+------+--------+----+----+----+------+
| 7369| SMITH|   JUNIOR|7902| 800|null|    20|
| 7499| ALLY|SALESWOMAN|7698|1600| 300|    30|
| 7521|  WARDY|SALESWOMAN|7698|1250| 500|    30|
| 7566| JONES| SUPERVISOR|7839|2975|null|    20|
| 7654|MARTINI|SALESWOMAN|7698|1250|1400|    30|
| 7698| BLAKE| SUPERVISOR|7839|2850|null|    30|
+-----+------+--------+----+----+----+------+

输出

+------+-----+-------+--------+
|deptno|JUNIOR|SUPERVISOR|SALESWOMAN|
+------+-----+-------+--------+
|    30| null|   2850|    4100|
|    20|  800|   2975|    null|
+------+-----+-------+--------+

代码

val spark: SparkSession = SparkSession
  .builder.master("local")
  .appName("MyGroup")
  .getOrCreate()
import spark.implicits._
    val client: DataFrame = spark.sparkContext.parallelize(
Seq((7369,"SMITH","JUNIOR",7902,800,null,20),(7499,"ALLY","SALESWOMAN",7698,1250,300,30),(7521,"WARDY","SALESWOMAN",7698,1250,500,30),(7566,"JONES","SUPERVISOR",7839,2975,null,20),(7654,"MARTINI","SALESWOMAN",7698,1250,1400,30),(7698,"BLAKE","SUPERVISOR",7839,2850,null,30))
).toDF("empno","ename","job","mgr","sal","comm","deptno")

val rd = client.groupBy("deptno").agg(sum("sal"),filter("job")).show()

谁能帮我最后一个逻辑。

【问题讨论】：

感谢 koiralo 的编辑。

标签： sql scala dataframe apache-spark

【解决方案1】：

您的groupBy/agg 是正确的，除了filter("job") 在agg() 中没有任何意义。在job 列上改用pivot，如下所示：

val df = Seq(
  (7369, "SMITH", "JUNIOR", 7902, 800, None, 20),
  (7499, "ALLY", "SALESWOMAN", 7698, 1600, Some(300), 30),
  (7521, "WARDY", "SALESWOMAN", 7698, 1250, Some(500), 30),
  (7566, "JONES", "SUPERVISOR", 7839, 2975, None, 20),
  (7654, "MARTINI", "SALESWOMAN", 7698, 1250, Some(1400), 30),
  (7698, "BLAKE", "SUPERVISOR", 7839, 2850, None, 30)
).toDF("empno", "ename", "job", "mgr", "sal", "comm", "deptno")

df.groupBy("deptno").pivot("job").agg(sum("sal")).show
// +------+------+----------+----------+
// |deptno|JUNIOR|SALESWOMAN|SUPERVISOR|
// +------+------+----------+----------+
// |    20|   800|      null|      2975|
// |    30|  null|      4100|      2850|
// +------+------+----------+----------+

【讨论】：