如何在python中迭代文件中的json项目并将其附加到列表中

Question

我是 json 项目操作的新手。目前，我正在做一个 NLP 项目。我正在使用 Spacy 构建 NER 模型。对于注释，我使用了 dataturks.com 注释服务。他们给出了一个格式化的 JSON 文件 示例 json 项目是这样的。准确地说，这个 json 项来自 CSV 文件

{"content": "Canada,Airdrie,Alberta,M1B 0V1 ,418-555-0122","annotation":[{"label":["Phone Number"],"points":[{"start":32,"end":43,"text":"418-555-0122"}]},{"label":["Postal Code"],"points":[{"start":23,"end":29,"text":"M1B 0V1"}]},{"label":["Province"],"points":[{"start":15,"end":21,"text":"Alberta"}]},{"label":["City"],"points":[{"start":7,"end":13,"text":"Airdrie"}]},{"label":["Country"],"points":[{"start":0,"end":5,"text":"Canada"}]}],"extras":null,"metadata":{"first_done_at":1545039231000,"last_updated_at":1545039231000,"sec_taken":50,"last_updated_by":"eODxmFU8wjPq8GJrmJtb0s7Wn9u1","status":"done","evaluation":"NONE"}}
{"content": "Canada,Barrie,Ontario,J2K 3C7 ,418-555-0135","annotation":[{"label":["Phone Number"],"points":[{"start":31,"end":42,"text":"418-555-0135"}]},{"label":["Postal Code"],"points":[{"start":22,"end":28,"text":"J2K 3C7"}]},{"label":["Province"],"points":[{"start":14,"end":20,"text":"Ontario"}]},{"label":["City"],"points":[{"start":7,"end":12,"text":"Barrie"}]},{"label":["Country"],"points":[{"start":0,"end":5,"text":"Canada"}]}],"extras":null,"metadata":{"first_done_at":1545157658000,"last_updated_at":1545157658000,"sec_taken":21,"last_updated_by":"eODxmFU8wjPq8GJrmJtb0s7Wn9u1","status":"done","evaluation":"NONE"}}
{"content": "Canada,Brandon,Manitoba,B1A 2X0 ,418-555-0171","annotation":[{"label":["Phone Number"],"points":[{"start":33,"end":44,"text":"418-555-0171"}]},{"label":["Postal Code"],"points":[{"start":24,"end":30,"text":"B1A 2X0"}]},{"label":["Province"],"points":[{"start":15,"end":22,"text":"Manitoba"}]},{"label":["City"],"points":[{"start":7,"end":13,"text":"Brandon"}]},{"label":["Country"],"points":[{"start":0,"end":5,"text":"Canada"}]}],"extras":null,"metadata":{"first_done_at":1545113770000,"last_updated_at":1545113770000,"sec_taken":27,"last_updated_by":"eODxmFU8wjPq8GJrmJtb0s7Wn9u1","status":"done","evaluation":"NONE"}}

我的代码 sn-p 是

trainingfilename="C:/Users/codemen/Desktop/Timeseries Analytics/Canadianinfo.json"

logging.basicConfig(level=logging.INFO)
def ConvertDataturkToSpacy(trainingfilename):

    try:
        trainingData=[]
        lines=[]
        # reading file  and  formating  part
        with open(trainingfilename,'r') as f:
            lines=f.readlines()
        for line in lines:
            data=json.loads(line)
            #wprint(data)
            text=data['content']
            print("Mytext",text)
            entities=[]
            #print('entties',entities)
            for annotation in data['annotation']:
                #print("Here is the thing")
                points=annotation['points'][0] #single point annotation part
                #print(point)
                labels=annotation['label']
                print(labels)
                #print("type",type(labels))
                if not isinstance(labels,list):#handling both list of labels or single label
                    labels=[labels]
                   # print("instance",labels)

                for label in labels:
                    #dataturks indices are inclusive but spacy indices are not so dealing with it by adding  with +1
                    #print("Test here")
                    #print ("label")
                    #print("priniting label")
                    #print(label)
                    #print(" inside type",type(label))
                   # print(points['start'],points["end"]+1,label)
                    entities.append((points["start"],points["end"]+1,label))
                    #entities.append({points['start'],points["end"]+1,label})
                    #print("MyEntities",entities)


            trainingData.append((text,{"entities":entities}))
            return trainingData
            #print("TrainingData",trainingData)
            #print("Datatype",type(trainingData))
        #return trainingData
    except Exception as e:

        logging.exception("Unable to process item" + trainingfilename +"\n"+ "errror ="+str(e))
        return None




TrainingData=ConvertDataturkToSpacy(trainingfilename)   

我已经注释掉了我想要的返回语句，如果我把它放在那里它会显示None type object not iterable error。所以为了测试目的，我在 for 循环中放了一个 return 语句，它实际上返回了一个列表，这是我文件中的第一个 json 项

[('Canada,Airdrie,Alberta,M1B 0V1 ,418-555-0122', {'entities': [(32, 44, 'Phone Number'), (23, 30, 'Postal Code'), (15, 22, 'Province'), (7, 14, 'City'), (0, 6, 'Country')]})]

我想要的列表是这样的，但是有 1000 个这样的列表元素。所以我认为我没有正确操作 JSON。请帮我解决这个问题 谢谢

Answer 1

将任务进一步划分可能有助于定位您的错误。换句话说，读取数据并从 json 转换为 dict 在概念上与阅读 dicts 和提取/重新格式化信息是分开的。您可以单独测试它们，例如确保您的 .json 实际加载到内存中。

对我来说，以下给出了您指示的输出并且没有错误（稍微修改您的代码但保持相同的逻辑）：

with open("places.json",'r') as fh:
    txt = fh.readlines()
    reps = [json.loads(l) for l in txt]

def extract_info(js):
  try:
    entities = []
    for e in js['annotation']:
        labs = e['label'] if isinstance(e['label'],list) else [e['label']]
        pts = e['points'][0]
        for lab in labs:
            entities.append((pts["start"],pts["end"]+1,lab))
    return (js['content'],{"entities" : entities})
  except Exception as e:
    print(e)
    return None

# can make this an explicit for loop if needed for debugging purposes
training_data = [extract_info(r) for r in reps]

[('Canada,Airdrie,Alberta,M1B 0V1 ,418-555-0122',
  {'entities': [(32, 44, 'Phone Number'),
    (23, 30, 'Postal Code'),
    (15, 22, 'Province'),
    (7, 14, 'City'),
    (0, 6, 'Country')]}),
 ('Canada,Barrie,Ontario,J2K 3C7 ,418-555-0135',
  {'entities': [(31, 43, 'Phone Number'),
    (22, 29, 'Postal Code'),
    (14, 21, 'Province'),
    (7, 13, 'City'),
    (0, 6, 'Country')]}),
 ('Canada,Brandon,Manitoba,B1A 2X0 ,418-555-0171',
  {'entities': [(33, 45, 'Phone Number'),
    (24, 31, 'Postal Code'),
    (15, 23, 'Province'),
    (7, 14, 'City'),
    (0, 6, 'Country')]})]

如果没有进一步的上下文，可能无法判断问题出在哪里（数据格式是否存在特殊之处？），因此可能只是在提取过程中考虑更多边缘情况。

FWIW，你提到的错误很容易识别：

for x in None: print(x) # TypeError: 'NoneType' object is not iterable

Answer 2

Pandas 有一些用于探索 json 的有用功能，假设您的 json 是有效的，请使用以下内容加载它：

   import pandas as pd

   with open(filepath, 'r') as jdata:
       jobj =  json.load(jdata)
       df = pd.read_json(jobj)
       print(df)

一旦有了 pandas 数据框，您就可以将 json 视为行和列的表 并且可以选择行和列的子集来隔离您需要的数据，或者如果没有其他方法可以更清楚地了解数据