使用 JSON 功能读取 CSV

Question

我正在尝试读取包含 JSON 功能的大型 CSV（location 此处）。首先，比如 100 行，文件如下所示：

Time,location,labelA,labelB
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},nan,nan

我跟着 this question 解析了 location 列。该解决方案基本上将帮助程序定义为：

def CustomParser(data):
    import json
    j1 = json.loads(data)
    return j1

然后

df=pd.read_csv('data.csv', nrows=100,converters={'location':CustomParser},header=0)

我收到以下与 JSON 格式有关的错误：

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Q1：如何将特征位置解析到新列上？

Q2（对于一般情况）：对于数据中的 nrows>100，最后的特征（labelA 和 labelB）也具有具有不同键和值的 JSON 格式。如何通过解析包括 JSON（甚至部分）的每个功能来读取整个 CSV？

Answer 1

修复文件：

• 很遗憾，该文件难以阅读，因为每一行都包含一个dict，其key-value 对以逗号分隔。
• 解决此问题的最简单方法是将每个 dict 之外的分隔符从 , 更改为 |。
• 以下代码将读取现有文件
  • 假设第一行是标题，使用.replace(',', '|')
  • 剩余的行将使用正则表达式替换, 之外的{}
  • 每一行都将写入一个新文件。

代码：

数据：

Time,location,labelA,labelB
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},{"ack":123,"bar":456},{"foo":123,"bar":456}
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},nan,nan
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},{"ack":123,"bar":456},{"foo":123,"bar":456}
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},nan,nan
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},{"ack":123,"bar":456},{"foo":123,"bar":456}
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},nan,nan
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},{"ack":123,"bar":456},{"foo":123,"bar":456}
2019-09-10,{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8},nan,nan

• Path.cwd() 假设 current working directory，如果不是这样：
  • Path('c:/some_path_to_my_file') / 'file_name.poo'可以使用
• pathlib 是标准库的一部分
• Python 3's pathlib Module: Taming the File System

文件修复：

import re
from pathlib import Path

p = Path.cwd() / 'test.csv'
p2 = Path.cwd() / 'test2.csv'

with p.open('r') as f:
    with p2.open('w') as f2:
        for cnt, line in enumerate(f):
            if cnt == 0:
                line = line.replace(',', '|')
            else:
                line = re.sub(r',(?=(((?!\}).)*\{)|[^\{\}]*$)', '|', line)
            f2.write(line)

新文件：

Time|location|labelA|labelB
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|{"ack":123,"bar":456}|{"foo":123,"bar":456}
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|nan|nan
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|{"ack":123,"bar":456}|{"foo":123,"bar":456}
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|nan|nan
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|{"ack":123,"bar":456}|{"foo":123,"bar":456}
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|nan|nan
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|{"ack":123,"bar":456}|{"foo":123,"bar":456}
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|nan|nan

解析新文件：

• 现在列将由.read_csv 正确分隔
• 但是，location、labelA 和 labelB 列是 str
  • 使用ast.literal_eval 转换为dict
  • literal_eval 无法在 nan 上工作，因此将 nan 替换为 {}
• for col in df.columns[1:]: 循环遍历每一列并且：
  • try-except 将捕获任何格式不正确的列
  • 将它们从str 转换为dict
  • 将keys 分隔成列
  • concats 现有数据框的列
  • drops老专栏

import pandas as pd
from ast import literal_eval

df = pd.read_csv('test2.csv', sep='|')
print(df)

       Time                                                             location                 labelA                 labelB
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}  {"ack":123,"bar":456}  {"foo":123,"bar":456}
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}                    NaN                    NaN
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}  {"ack":123,"bar":456}  {"foo":123,"bar":456}
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}                    NaN                    NaN
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}  {"ack":123,"bar":456}  {"foo":123,"bar":456}
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}                    NaN                    NaN
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}  {"ack":123,"bar":456}  {"foo":123,"bar":456}
 2019-09-10  {"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}                    NaN                    NaN


for col in df.columns[1:]:
    try:
        df[col].fillna('{}', inplace=True)
        df[col] = df[col].apply(literal_eval)
        df = pd.concat([df, df[col].apply(pd.Series)], axis=1)
        df.drop(columns=[col], inplace=True)
    except (SyntaxError, ValueError) as e:
        print(f'{col}: {e}')


print(df)

       Time   lng    alt        time  error   lat    ack    bar    foo    bar
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8  123.0  456.0  123.0  456.0
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8    NaN    NaN    NaN    NaN
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8  123.0  456.0  123.0  456.0
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8    NaN    NaN    NaN    NaN
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8  123.0  456.0  123.0  456.0
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8    NaN    NaN    NaN    NaN
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8  123.0  456.0  123.0  456.0
 2019-09-10  12.9  413.0  2019-09-10    7.0  17.8    NaN    NaN    NaN    NaN

文字评估注释：

• Pandas 提供多种形式的数据导入方法，例如dict 或list。
• 但是，read_csv 不能很好地解释容器（例如 dict），它们被解释为字符串，除非您指定 converters 参数（pd.read_csv('test3.csv', sep='|', converters={'a': literal_eval})。
• literal_eval 不适用于同时包含容器和 strings 或 NaN 的列，除非 string 只是数字（例如“8654”）
• 上面的部分代码，首先将所有nan替换为{}，这样literal_eval就不会出错了。
• 给出以下混合列​​示例：

column_a
{"ack":123,"bar":456}
some string
{"ack":123,"bar":456}
some string
{"ack":123,"bar":456}
some string

• literal_eval 将抛出 ValueError: malformed node or string:
  • 这两种解决方案之间的区别在于，另一种解决方案固定一列，而此解决方案的实施方式是固定所有列并消除了仅读取前 100 行的必要性。
  • 如果是dicts，您可以放弃修复所有列的循环，只修复location 列。使用以下代码：

df['location'] = df['location'].apply(literal_eval)
df = pd.concat([df, df['location'].apply(pd.Series)], axis=1)

注意实际数据：

• location 列的格式不正确
  • '{"lng":12.9975201,alt:413.0,"time:""2019-09-10T12:09:58Z""",error:7.0,lat:47.8258582}'
• 这是预期的形式：
  • '{"lng":12.9975201,"alt":413.0,"time":"2019-09-10T12:09:58Z","error":7.0,"lat":47.8258582}'

修复location 列：

• location 列在真实数据中为Position

def fix_pos(x):
    word_dict = {'alt': '"alt"',
                 '"time:"': '"time":',
                 '"",error:': ',"error":',
                 'lat': '"lat"'}
    for k, v in word_dict.items():
        x = x.replace(k, v)
    return x

df.Position = df.Position.apply(lambda x: fix_pos(x))

• 对真实数据文件使用以下循环。
• Zeit, device, Text & Type 不需要处理
• Position 位于index 4。

for col in df.columns[4:]:
    try:
        df[col].fillna('{}', inplace=True)
        df[col] = df[col].apply(literal_eval)
        df = pd.concat([df, df[col].apply(pd.Series)], axis=1)
        df.drop(columns=[col], inplace=True)
    except (SyntaxError, ValueError) as e:
        print(f'{col}: {e}')

• 将literal_eval 应用于所有列的循环已更新为try-except
  • 如果有exception，则会打印出column 名称和错误消息。
  • 真实数据共有64列，大部分为Furchtbar。

错误：

• 这些是提供的csv 文件中所有列的错误。

device: unexpected EOF while parsing (<unknown>, line 1)
Text: malformed node or string: <_ast.Name object at 0x00000203B8473C08>
Typ: malformed node or string: <_ast.Name object at 0x00000203BE217E08>
Data: unexpected EOF while parsing (<unknown>, line 1)
Data1: invalid syntax (<unknown>, line 1)
Data2: invalid syntax (<unknown>, line 1)
Unnamed: 8: invalid syntax (<unknown>, line 1)
Unnamed: 9: unexpected EOF while parsing (<unknown>, line 1)
Unnamed: 10: invalid syntax (<unknown>, line 1)
Unnamed: 11: unexpected EOF while parsing (<unknown>, line 1)
Unnamed: 12: invalid syntax (<unknown>, line 1)
Unnamed: 13: invalid syntax (<unknown>, line 1)
Unnamed: 14: invalid syntax (<unknown>, line 1)
Unnamed: 15: invalid syntax (<unknown>, line 1)
Unnamed: 16: invalid syntax (<unknown>, line 1)
Unnamed: 17: invalid syntax (<unknown>, line 1)
Unnamed: 18: invalid syntax (<unknown>, line 1)
Unnamed: 19: invalid syntax (<unknown>, line 1)
Unnamed: 20: invalid syntax (<unknown>, line 1)
Unnamed: 21: unexpected EOF while parsing (<unknown>, line 1)
Unnamed: 22: invalid syntax (<unknown>, line 1)
Unnamed: 23: invalid syntax (<unknown>, line 1)
Unnamed: 24: invalid syntax (<unknown>, line 1)
Unnamed: 25: invalid syntax (<unknown>, line 1)
Unnamed: 26: invalid syntax (<unknown>, line 1)
Unnamed: 27: invalid syntax (<unknown>, line 1)

Answer 2

这里的问题是json 字符串中的逗号被视为分隔符。您应该修改输入数据（如果您无法直接访问该文件，您始终可以先使用open 将内容读入字符串列表）。

您可以尝试以下一些修改选项：

选项 1：用单引号引用 json 字符串

使用单引号（或其他不会出现在数据中的字符）作为json 字符串的引号字符。

>> cat data.csv
Time,location,labelA,labelB
2019-09-10,'{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}',nan,nan

然后在读取数据时使用quotechar="'"：

import pandas as pd
import json

df=pd.read_csv('data.csv', converters={'location':json.loads}, header=0, quotechar="'")

选项 2：用双引号引用 json 字符串并转义

如果不能使用单引号，你可以实际上使用双引号作为quotechar，只要你将json字符串中的引号转义即可：

>> cat data.csv
Time,location,labelA,labelB
2019-09-10,"{""lng"":12.9,""alt"":413.0,""time"":""2019-09-10"",""error"":7.0,""lat"":17.8}",nan,nan

请注意，这现在与您链接的问题的格式相匹配。

df=pd.read_csv('data.csv', converters={'location':json.loads}, header=0, quotechar='"')

选项 3：更改分隔符

使用不同的字符，例如| 作为分隔符

>> cat data.csv
Time|location|labelA|labelB
2019-09-10|{"lng":12.9,"alt":413.0,"time":"2019-09-10","error":7.0,"lat":17.8}|nan|nan

现在使用sep 参数来指定新的分隔符：

df=pd.read_csv('data.csv', converters={'location':json.loads}, header=0, sep="|")

---

这些方法中的每一个都产生相同的输出：

print(df)
#   Time        location                                            labelA  labelB
#0  2019-09-10  {u'lat': 17.8, u'lng': 12.9, u'error': 7.0, u'...   NaN     NaN

完成后，您可以使用Flatten JSON column in a Pandas DataFrame 中描述的方法之一扩展location 列

new_df = df.join(pd.io.json.json_normalize(df["location"])).drop(["location"], axis=1)
print(new_df)
#   Time        labelA  labelB  alt    error  lat   lng   time
#0  2019-09-10  NaN     NaN     413.0  7.0    17.8  12.9  2019-09-10