如何在不创建额外行的情况下合并 2 个数据框？

Question

我有 2 个数据框，我想做 .filter($"item" === "a") 同时将“S/N”保持在数字中。

我尝试了以下方法，但当我使用联合时，它最终得到了额外的行。有没有办法在不创建额外行的情况下合并 2 个数据框？

var DF1 = Seq(
  ("1","a",2),
  ("2","a",3),
  ("3","b",3),
  ("4","b",4),
  ("5","a",2)).
  toDF("S/N","item", "value")

var DF2 = Seq(
  ("1","a",2),
  ("2","a",3),
  ("3","b",3),
  ("4","b",4),
  ("5","a",2)).
  toDF("S/N","item", "value")
DF2 = DF2.filter($"item"==="a")

DF3=DF1.withColumn("item",lit(0)).withColumn("value",lit(0))

DF1.show()
+---+----+-----+
|S/N|item|value|
+---+----+-----+
|  1|   a|    2|
|  2|   a|    3|
|  3|   b|    3|
|  4|   b|    4|
|  5|   a|    2|
+---+----+-----+

DF2.show()
+---+----+-----+
|S/N|item|value|
+---+----+-----+
|  1|   a|    2|
|  2|   a|    3|
|  5|   a|    2|
+---+----+-----+

DF3.show()
+---+----+-----+
|S/N|item|value|
+---+----+-----+
|  1|   0|    0|
|  2|   0|    0|
|  3|   0|    0|
|  4|   0|    0|
|  5|   0|    0|
+---+----+-----+

DF2.union(someDF3).show()
+---+----+-----+
|S/N|item|value|
+---+----+-----+
|  1|   a|    2|
|  2|   a|    3|
|  5|   a|    2|
|  1|   0|    0|
|  2|   0|    0|
|  3|   0|    0|
|  4|   0|    0|
|  5|   0|    0|
+---+----+-----+

Answer 1

使用过滤后的数据框将您的 S/N 左外连接，然后使用 coalesce 去除空值：

val DF3 = DF1.select("S/N")

val DF4 = (DF3.join(DF2, Seq("S/N"), joinType="leftouter")
              .withColumn("item", coalesce($"item", lit(0)))
              .withColumn("value", coalesce($"value", lit(0))))
DF4.show
+---+----+-----+
|S/N|item|value|
+---+----+-----+
|  1|   a|    2|
|  2|   a|    3|
|  3|   0|    0|
|  4|   0|    0|
|  5|   a|    2|
+---+----+-----+