如何使用正则表达式从 pyspark 数据框列中删除双引号答案

【问题标题】：How to remove double quotes from a pyspark dataframe column using regex如何使用正则表达式从 pyspark 数据框列中删除双引号
【发布时间】：2021-09-04 10:57:53
【问题描述】：

我正在从具有 json 数据的文本文件创建一个数据框（df）。创建后的数据框如下所示。

+------------------------------------------------------------------------------------+
|data                                                                                |
+------------------------------------------------------------------------------------+
|"{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PARC PARQUE","State":"PR"}"        |
|"{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PASEO COSTA DEL SUR","State":"PR"}"|
+------------------------------------------------------------------------------------+

我想去掉列数据开头和结尾的双引号。所以最终的数据框应该是这样的

+------------------------------------------------------------------------------------+
|data                                                                                |
+------------------------------------------------------------------------------------+
|{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PARC PARQUE","State":"PR"}          |
|{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PASEO COSTA DEL SUR","State":"PR"}  |
+------------------------------------------------------------------------------------+

以下是我编写的用于从开头删除双引号的代码

df = df.withColumn('data1', F.regexp_replace("data",'^\"{\"','{\"'))

但我收到此错误

^"{" ^ 在 java.util.regex.Pattern.error(Pattern.java:1957)

你能帮我解决这个问题吗？

【问题讨论】：

标签： json dataframe apache-spark pyspark

【解决方案1】：

您只需要稍微调整一下您的正则表达式。不需要转义引号，但需要转义大括号：

df2 = df.withColumn('data1', F.regexp_replace("data",'^"\{"','{"'))

df2.show(truncate=False)
+------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+
|data                                                                                |data1                                                                              |
+------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+
|"{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PARC PARQUE","State":"PR"}"        |{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PARC PARQUE","State":"PR"}"        |
|"{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PASEO COSTA DEL SUR","State":"PR"}"|{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PASEO COSTA DEL SUR","State":"PR"}"|
+------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------+

【讨论】：