pandas to_json() 多余的反斜杠

Question

我有一个“.csv”文件，其中包含有关电影的数据，我正在尝试将其重新格式化为 JSON 文件以在 MongoDB 中使用它。所以我将该 csv 文件加载到 pandas DataFrame，然后使用 to_json 方法将其写回。 这是 DataFrame 中的一行的样子：

In [43]: result.iloc[0]
Out[43]: 
title                                                      Avatar
release_date                                                 2009
cast            [{"cast_id": 242, "character": "Jake Sully", "...
crew            [{"credit_id": "52fe48009251416c750aca23", "de...
Name: 0, dtype: object

但是当pandas写回来的时候，就变成了这样：

{   "title":"Avatar",
    "release_date":"2009",
    "cast":"[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"credit_id\": \"5602a8a7c3a3685532001c9a\", \"gender\": 2,...]",
    "crew":"[{\"credit_id\": \"52fe48009251416c750aca23\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1721,...]"
}

如您所见，“cast”和“crew”是列表，它们有大量多余的反斜杠。这些反斜杠出现在 MongoDB 集合中，无法从这两个字段中提取数据。

除了将\" 替换为" 之外，我该如何解决这个问题？

P.S.1：这就是我将 DataFrame 保存为 JSON 的方式：

result.to_json('result.json', orient='records', lines=True)

更新 1： 显然熊猫做得很好，问题是由原始 csv 文件引起的。 以下是它们的样子：

movie_id,title,cast,crew
19995,Avatar,"[{""cast_id"": 242, ""character"": ""Jake Sully"", ""credit_id"": ""5602a8a7c3a3685532001c9a"", ""gender"": 2, ""id"": 65731, ""name"": ""Sam Worthington"", ""order"": 0}, {""cast_id"": 3, ""character"": ""Neytiri"", ""credit_id"": ""52fe48009251416c750ac9cb"", ""gender"": 1, ""id"": 8691, ""name"": ""Zoe Saldana"", ""order"": 1}, {""cast_id"": 25, ""character"": ""Dr. Grace Augustine"", ""credit_id"": ""52fe48009251416c750aca39"", ""gender"": 1, ""id"": 10205, ""name"": ""Sigourney Weaver"", ""order"": 2}, {""cast_id"": 4, ""character"": ""Col. Quaritch"", ""credit_id"": ""52fe48009251416c750ac9cf"", ""gender"": 2, ""id"": 32747, ""name"": ""Stephen Lang"", ""order"": 3},...]"

我试图用" 替换""（我真的很想避免这种黑客攻击）：

sed -i 's/\"\"/\"/g'

当然，当再次将其读取为 csv 时，它会导致某些数据行出现问题：

ParserError: Error tokenizing data. C error: Expected 1501 fields in line 4, saw 1513

因此我们可以得出结论，进行这种盲目更换是不安全的。有什么想法吗？

P.S.2：我使用的是 kaggle 的 5000 部电影数据集：https://www.kaggle.com/carolzhangdc/imdb-5000-movie-dataset

Answer 1

Pandas 正在转义 " 字符，因为它认为 json 列中的值是文本。要获得所需的行为，只需将 json 列中的值解析为 json。

让文件 data.csv 具有以下内容（引号已转义）。

# data.csv
movie_id,title,cast
19995,Avatar,"[{""cast_id"": 242, ""character"": ""Jake Sully"", ""credit_id"": ""5602a8a7c3a3685532001c9a"", ""gender"": 2, ""id"": 65731, ""name"": ""Sam Worthington"", ""order"": 0}, {""cast_id"": 3, ""character"": ""Neytiri"", ""credit_id"": ""52fe48009251416c750ac9cb"", ""gender"": 1, ""id"": 8691, ""name"": ""Zoe Saldana"", ""order"": 1}, {""cast_id"": 25, ""character"": ""Dr. Grace Augustine"", ""credit_id"": ""52fe48009251416c750aca39"", ""gender"": 1, ""id"": 10205, ""name"": ""Sigourney Weaver"", ""order"": 2}, {""cast_id"": 4, ""character"": ""Col. Quaritch"", ""credit_id"": ""52fe48009251416c750ac9cf"", ""gender"": 2, ""id"": 32747, ""name"": ""Stephen Lang"", ""order"": 3}]"

将其读入数据帧，然后应用 json.loads 函数并以 json 格式写入文件。

df = pd.read_csv('data.csv')
df.cast = df.cast.apply(json.loads)
df.to_json('data.json', orient='records', lines=True)

输出是格式正确的 json（我添加了额外的换行符）

# data.json
{"movie_id":19995,
 "title":"Avatar",
 "cast":[{"cast_id":242,"character":"Jake Sully","credit_id":"5602a8a7c3a3685532001c9a","gender":2,"id":65731,"name":"Sam Worthington","order":0},
         {"cast_id":3,"character":"Neytiri","credit_id":"52fe48009251416c750ac9cb","gender":1,"id":8691,"name":"Zoe Saldana","order":1},
         {"cast_id":25,"character":"Dr. Grace Augustine","credit_id":"52fe48009251416c750aca39","gender":1,"id":10205,"name":"Sigourney Weaver","order":2},
         {"cast_id":4,"character":"Col. Quaritch","credit_id":"52fe48009251416c750ac9cf","gender":2,"id":32747,"name":"Stephen Lang","order":3}]
}

Answer 2

我遇到了同样的问题：解决方案分三步

1- 数据框形式 csv 或在我的情况下来自 xlsx：

 excel_df= pd.read_excel(dataset ,sheet_name=my_sheet_name)

2- 转换为 json（如果您的数据中有日期）

json_str = excel_df.to_json(orient='records' ,date_format='iso')

3-最重要的是：json.loads ****就是这样！

parsed = json.loads(json_str)

4-（兼任）您可以编写或发送 json 文件： 例如：本地写

with open(out, 'w') as json_file:
    json_file.write(json.dumps({"data": parsed}, indent=4 ))

更多信息： https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_json.html