【发布时间】：2019-12-04 16:04:39
【问题描述】：
我有这个数据集。为此，我需要在 scala 中创建一个 sparkdataframe。此数据是 csv 文件中的一列。列名是数据头
数据头
"{""date_time"":""1999/05/22 03:03:07.011"",""cust_id"":""cust1"",""timestamp"":944248234000,""msgId"":""113"",""activityTimeWindowMilliseconds"":20000,""ec"":""event1"",""name"":""ABC"",""entityId"":""1001"",""et"":""StateChange"",""logType"":""type123,""lastActivityTS"":944248834000,""sc_id"":""abc1d1c9"",""activityDetectedInLastTimeWindow"":true}"
"{""date_time"":""1999/05/23 03:03:07.011"",""cust_id"":""cust1"",""timestamp"":944248234000,""msgId"":""114"",""activityTimeWindowMilliseconds"":20000,""ec"":""event2"",""name"":""ABC"",""entityId"":""1001"",""et"":""StateChange"",""logType"":""type123,""lastActivityTS"":944248834000,""sc_id"":""abc1d1c9"",""activityDetectedInLastTimeWindow"":true}"
我能够读取 csv 文件 -
val df_tmp = spark
.read
.format("com.databricks.spark.csv")
.option("header","true")
.option("quoteMode", "ALL")
.option("delimiter", ",")
.option("escape", "\"")
//.option("inferSchema","true")
.option("multiline", "true")
.load("D:\\dataFile.csv")
我尝试将数据拆分为数据框中的单独列，但没有成功。
我在数据中注意到的一件事是键和值都用双双引号括起来""key1"":""value1""
【问题讨论】：
-
在这种情况下“没有成功”是什么意思？
-
我能够读取 csv
df_tmp: org.apache.spark.sql.DataFrame = [text: string]df_tmp.show(false)|text |{"date_time":"1999/05/22 03:03:07.011","cust_id":"cust1", "timestamp":944248234000,"msgId":"113","activityTimeWindowMilliseconds":20000,"ec":"event1","name":"ABC","entityId":"1001","et":"StateChange ","logType":"type123,"lastActivityTS":944248834000,"sc_id":"abc1d1c9","activityDetectedInLastTimeWindow":true}| -
请编辑您的问题以包含您期望的输出示例。我有一些想法可以帮助你，但我仍然不清楚你想要完成什么。
-
因为它是一个 json 字符串，所以我解析并应用了正则表达式将其放入数据框列中
标签： scala apache-spark