当逗号出现在 pyspark RDD 中的引号之间时忽略逗号答案

【问题标题】：Ignore comma when it appears between quotation marks in pyspark RDD当逗号出现在 pyspark RDD 中的引号之间时忽略逗号
【发布时间】：2020-12-17 22:10:35
【问题描述】：

我已将一个 csv 文件加载到 RDD 中，如下所示：

['"3331/587","Sub,Metro","1235","1000"',
'"1234/232","City","8479","2000"',
'"5987/215","Sub,Metro","1111","Unknown"',
'"8794/215","Sub,Metro","1112","1000"',
'"1254/951","City","6598","XXXX"',
'"1584/951","City","1548","Unknown"',
'"1833/331","Sub,Metro","1009","2000"',
'"2213/987","City","1197", ']

我最终想要实现的是

[["3331/587","Sub,Metro","1235","1000"],
["1234/232","City","8479","2000"],
["5987/215","Sub,Metro","1111","Unknown"],
["8794/215","Sub,Metro","1112","1000"],
["1254/951","City","6598","XXXX"],
["1584/951","City","1548","Unknown"],
["1833/331","Sub,Metro","1009","2000"],
["2213/987","City","1197", ]]

如果我使用此代码：

sc.textFile(file).map(lambda l: l.replace(r'"', '').split(','))

它还在值内用逗号分隔 ("Sub,Metro")

用逗号分割时如何自动忽略“”之间的所有逗号？

【问题讨论】：

标签： regex csv apache-spark pyspark rdd

【解决方案1】：

这是我的正则表达式示例。

import re

rdd = sc.parallelize(['"3331/587","Sub,Metro","1235","1000"',
'"1234/232","City","8479","2000"',
'"5987/215","Sub,Metro","1111","Unknown"',
'"8794/215","Sub,Metro","1112","1000"',
'"1254/951","City","6598","XXXX"',
'"1584/951","City","1548","Unknown"',
'"1833/331","Sub,Metro","1009","2000"',
'"2213/987","City","1197", '])

rdd.map(lambda l: re.split(r'[^A-z],[^A-z]', re.sub(r'(^["])|(["]$)','', l))).collect()

[['3331/587', 'Sub,Metro', '1235', '1000'],
 ['1234/232', 'City', '8479', '2000'],
 ['5987/215', 'Sub,Metro', '1111', 'Unknown'],
 ['8794/215', 'Sub,Metro', '1112', '1000'],
 ['1254/951', 'City', '6598', 'XXXX'],
 ['1584/951', 'City', '1548', 'Unknown'],
 ['1833/331', 'Sub,Metro', '1009', '2000'],
 ['2213/987', 'City', '1197', '']]

【讨论】：