如何用分隔符在火花中爆炸

Question

我有一张桌子：

id itemNames 优惠券 有 1 个项目（foo bar）可用，肥皂真 2 项（条）可用 false 3 香皂、洗发水假 4 项 (foo bar, bar) 可用 true 5 项 (foo bar, bar) 可用, (肥皂, 洗发水) true 6 空假

我想把这个炸成

id itemNames 优惠券 1 个项目（foo bar）可用 true
1 肥皂真 2 项（条）可用 false 3 肥皂是假的 3羞耻假 4 项 (foo bar, bar) 可用 true 5 项 (foo bar, bar) 可用 true 6（肥皂、洗发水）是的
6空真

当我这样做时：

 df.withColumn("itemNames", explode(split($"itemNames", "[,]")))

我得到了：

itemNames                                          coupons
item (foo bar) is available                        true       
soaps                                              true 
item (bar) is available                            false
soaps                                              false
shampoo                                            false
item (foo bar,                                     true
bar) is available                                  true 
(soap,                                             true    
shampoo)                                           true

谁能告诉我我做错了什么，我该如何纠正？这里常见的一种模式是逗号出现在 () 中。

Answer 1

使用 UDF 并受到 Regex to match only commas not in parentheses? 的启发：

val df = List(
  ("item (foo bar) is available, soaps", true),
  ("item (bar) is available", false),
  ("soaps, shampoo", false),
  ("item (foo bar, bar) is available", true),
  ("item (foo bar, bar) is available, (soap, shampoo)", true)
).
  toDF("itemNames", "coupons")
df.show(false)

val regex = Pattern.compile(
  ",         # Match a comma\n" +
    "(?!       # only if it's not followed by...\n" +
    " [^(]*    #   any number of characters except opening parens\n" +
    " \\)      #   followed by a closing parens\n" +
    ")         # End of lookahead",
  Pattern.COMMENTS)

val customSplit = (value: String) => regex.split(value)
val customSplitUDF = udf(customSplit)
val result = df.withColumn("itemNames", explode(customSplitUDF($"itemNames")))
result.show(false)

输出是：

+--------------------------------+-------+
|itemNames                       |coupons|
+--------------------------------+-------+
|item (foo bar) is available     |true   |
| soaps                          |true   |
|item (bar) is available         |false  |
|soaps                           |false  |
| shampoo                        |false  |
|item (foo bar, bar) is available|true   |
|item (foo bar, bar) is available|true   |
| (soap, shampoo)                |true   |
+--------------------------------+-------+

如果需要“trim”，可以轻松添加到“customSplit”中。

Answer 2

您的问题没有将字符串向后拆分的模式。以下是一种解决方法，适用于这种特殊情况。我使用后向操作除以“可用”。在你的数据框中尝试这个爆炸

scala> "item (foo bar) is available, soaps".split("(?<=available),")
res41: Array[String] = Array(item (foo bar) is available, " soaps")

scala> "item (foo bar) is available, soaps".split("(?<=available),").length
res42: Int = 2

scala> "item (foo bar, bar) is available".split("(?<=available),")
res44: Array[String] = Array(item (foo bar, bar) is available)

scala> "item (foo bar, bar) is available".split("(?<=available),").length
res45: Int = 1

EDIT1

scala> "item (foo bar, bar) is empty, (soap, shampoo)".split("(?<=available|empty),").length
res1: Int = 2

scala>