将嵌套的 json/dict 转换为元组格式时出现问题？

Question

更新

考虑以下dict。如何提取符合以下条件的 4 元组： lema、original_form、tag，当且仅当它存在 id。到目前为止，这是我尝试过的：

def gettuples(data, level = 0):
    if isinstance(data, dict):
        if 'semtheme_list' in data:
            print(data['semtheme_list'][0])
            yield data['semtheme_list'][0]

        elif 'analysis_list' in data:
            print(data['analysis_list'][0])
            yield data['analysis_list'][0]

        for val in data.values():
            yield from gettuples(val)

    elif isinstance(data, list):
        for val in data:
            yield from gettuples(val)

通过上述函数，我得到以下（*）：

{'lemma': '*', 'tag': 'Z-----------', 'original_form': "Robert Downey Jr has topped Forbes magazine's annual list John Deere"}
{'lemma': 'Robert Downey Jr', 'tag': 'GNUS3S--', 'original_form': 'Robert Downey Jr'}
{'sense_id_list': [{'sense_id': '__12123288058840445720'}], 'lemma': 'Robert Downey Jr', 'tag': 'NPUU-N-', 'original_form': 'Robert Downey Jr'}
{'lemma': 'top', 'tag': 'VI-S3PPA-N-N9', 'original_form': 'has topped'}
{'lemma': 'John Deere', 'tag': 'GN-S3D--', 'original_form': "Forbes magazine's annual list John Deere"}
{'lemma': 'magazine', 'tag': 'GN-S3---', 'original_form': 'Forbes magazine'}
{'sense_id_list': [{'sense_id': 'db0f9829ff'}], 'lemma': 'Forbes', 'tag': 'NP-S-N-', 'original_form': 'Forbes'}
{'type': 'Top>SocialSciences>Economy', 'id': 'ODTHEME_ECONOMY'}

这与我正在寻找的 4 元组非常相似，用于 (**)：

 [[['Z-----------', "Robert Downey Jr has topped Forbes magazine's annual list John Deere", '*'], ['GNUS3S--', 'Robert Downey Jr', 'Robert Downey Jr'], ['NPUU-N-', 'Robert Downey Jr', 'Robert Downey Jr'], ['VI-S3PPA-N-N9', 'has topped', 'top'], ['GN-S3D--', "Forbes magazine's annual list John Deere", 'John Deere'], ['GN-S3---', 'Forbes magazine', 'magazine'], ['NP-S-N-', 'Forbes', 'Forbes'], ['NC-S-N5', 'magazine', 'magazine'], ['WN-', "'s", "'s"], ['GN-S3---', 'annual list John Deere', 'John Deere'], ['GN-S3---', 'annual list', 'list'], ['AP-N5', 'annual', 'annual'], ['NC-S-N5', 'list', 'list'], ['GN-S3Y--', 'John Deere', 'John Deere'], ['NP-S-N-', 'John Deere', 'John Deere']]]

但是entity_listid:

 entity_list: [{ form: "John Deere", official_form: "Deere & Company", id: "d5250a54a8", sementity: { class: "instance", fiction: "nonfiction", id: "ODENTITY_INDUSTRIAL_COMPANY", type: "Top>Organization>Company>IndustrialCompany" 
}

然后，当我打印时：

result = [['lema:',obj['lemma'], 'original_form', obj['original_form'], 'tag:',obj['tag']] for obj in gettuples(json_data)]

print(result)

我收到了这个错误：

  File "/Users/user/PycharmProjects/Tests/test.py", line 51, in pos_tag2
    result = [['lema:',obj['lemma'], 'original_form', obj['original_form'], 'tag:',obj['tag']] for obj in gettuples(json_data)]
  File "/Users/user/PycharmProjects/Tests/test.py", line 51, in <listcomp>
    result = [['lema:',obj['lemma'], 'original_form', obj['original_form'], 'tag:',obj['tag']] for obj in gettuples(json_data)]
KeyError: 'lemma'

所以，我的问题是：我怎样才能从 (*) 获得像 (**) 这样的 4 元组格式？或者我应该遵循哪种其他方法来提取符合 lema 的 4 元组， original_form，tag，当且仅当它存在 id?。

更新 2

另外，我尝试过的另一件事是json_normalize：

在：

from pandas.io.json import json_normalize
df = json_normalize(request, ['token_list',['token_list']])
df = pd.DataFrame(df)
df

输出：

    affected_by_negation    analysis_list   endp    form    id  inip    quote_level     separation  style   token_list  type
0   no  [{'lemma': '*', 'tag': 'Z-----------', 'origin...   4   Deere   6   0   0   _   {'isTitle': 'no', 'isItalics': 'no', 'isUnderl...   [{'form': 'Deere', 'analysis_list': [{'lemma':...   phrase

然后：

df_clean =  df.drop(df.columns[[0, 2,4, 5, 6, 7, 8, 10]], axis=1)
df_clean
list(df_clean.itertuples(index=False))

输出：

[Pandas(analysis_list=[{'lemma': '*', 'tag': 'Z-----------', 'original_form': 'Deere'}], form='Deere', token_list=[{'form': 'Deere', 'analysis_list': [{'lemma': 'Edere', 'tag': 'GN-S3---', 'original_form': 'Deere'}, {'lemma': 'deer', 'tag': 'GN-S3---', 'original_form': 'Deere'}, {'lemma': 'Edere', 'tag': 'GN-P3---', 'original_form': 'Deere'}, {'lemma': 'deer', 'tag': 'GN-P3---', 'original_form': 'Deere'}, {'lemma': 'Edere', 'tag': 'GNFU3---', 'original_form': 'Deere'}], 'head': '1', 'separation': '_', 'affected_by_negation': 'no', 'endp': '4', 'type': 'phrase', 'style': {'isTitle': 'no', 'isItalics': 'no', 'isUnderlined': 'no', 'isBold': 'no'}, 'id': '5', 'inip': '0', 'token_list': [{'form': 'Deere', 'affected_by_negation': 'no', 'sense_list': [{'id': '228eaef205', 'info': 'sementity/class=class@fiction=nonfiction@id=ODENTITY_MAMMAL@type=Top>LivingThing>Animal>Vertebrate>Mammal\tsemld_list=sumo:Mammal\tsemtheme_list/id=ODTHEME_ZOOLOGY@type=Top>NaturalSciences>Zoology', 'form': 'deer'}, {'id': 'e7c6da7489', 'info': 'sementity/class=instance@fiction=nonfiction@id=ODENTITY_FIRST_NAME@type=Top>Person>FirstName\tsemld_list=sumo:FirstName', 'form': 'Edere'}], 'separation': '_', 'style': {'isTitle': 'no', 'isItalics': 'no', 'isUnderlined': 'no', 'isBold': 'no'}, 'id': '1', 'inip': '0', 'topic_list': {'entity_list': [{'semld_list': ['sumo:FirstName'], 'form': 'Edere', 'sementity': {'id': 'ODENTITY_FIRST_NAME', 'class': 'instance', 'fiction': 'nonfiction', 'type': 'Top>Person>FirstName'}, 'id': 'e7c6da7489'}], 'concept_list': [{'semld_list': ['sumo:Mammal'], 'form': 'deer', 'semtheme_list': [{'id': 'ODTHEME_ZOOLOGY', 'type': 'Top>NaturalSciences>Zoology'}], 'sementity': {'id': 'ODENTITY_MAMMAL', 'class': 'class', 'fiction': 'nonfiction', 'type': 'Top>LivingThing>Animal>Vertebrate>Mammal'}, 'id': '228eaef205'}]}, 'analysis_list': [{'lemma': 'Edere', 'sense_id_list': [{'sense_id': 'e7c6da7489'}], 'tag': 'NPFS-N-', 'original_form': 'Edere', 'check_info': {'form_list': [{'form': 'Edere'}], 'tag': '6'}}, {'lemma': 'deer', 'sense_id_list': [{'sense_id': '228eaef205'}], 'tag': 'NC-S-N2', 'original_form': 'deer', 'check_info': {'form_list': [{'form': 'deer'}], 'tag': '6'}}, {'lemma': 'deer', 'sense_id_list': [{'sense_id': '228eaef205'}], 'tag': 'NC-P-N2', 'original_form': 'deer', 'check_info': {'form_list': [{'form': 'deer'}], 'tag': '6'}}], 'quote_level': '0', 'endp': '4'}], 'quote_level': '0'}])]

不过，我在访问列表的特定值时遇到了问题。另一个可能的解决方案可能是使用 pandas ......伙计们知道怎么做吗？

Answer 1

以下代码应该可以满足您的需求。这不是最优雅的方法，但希望它很清楚。

import yaml
from pprint import pprint

with open('json_dict.json', 'rU') as f:
    data = yaml.load(f)

results = []
sementity_map = {}

def extract_analysis(l):
    for d in l:
        out = {
            'lemma': d['lemma'],
            'original_form': d['original_form'],
            'tag': d['tag']
        }

        if 'sense_id_list' in d:
            out['id'] = d['sense_id_list'][0]['sense_id']

        results.append( out )

def extract_entities(l):
    for d in l:
        if 'sementity' in d and 'id' in d['sementity']:
            sementity_map[ d['id'] ] = d['sementity']['id']


def find_analysis_and_entities(d):
    if type(d) != dict:  # Added for non-dict values
        return # Fail

    for k, v in d.items():
        if type(v) == list:
            if k == 'analysis_list':
                extract_analysis(v)
            elif k == 'entity_list':
                extract_entities(v)
            else:
                for do in v:
                    find_analysis_and_entities(do)
        else:
            find_analysis_and_entities(v)

def apply_entities(e, m):
    for d in e:
        if 'id' in d:
            if d['id'] in sementity_map:
                d['id'] = sementity_map[ d['id'] ]
            else:
                del d['id']

find_analysis_and_entities(data)
apply_entities(results, sementity_map)                

pprint(results)

对于语义 id，我们保留一个单独的地图字典，并在初始查找运行后应用它。第一个 find 用于构建带有裸 id 的结果和语义实体映射。

部分问题（我认为）源于这样一个事实，即在找到必须应用的位置之前，您无法确定是否找到/传递了匹配的语义实体 id（使用 dicts 没有帮助)。

这里我们只在找到它们时应用 id 映射，否则我们删除该 id 字段。例如，a0a1a5401f 和 __12123288058840445720 都没有列在 entity_list 块中，因此会从 results 中删除。

上面的输出与您的示例输入文件是：

[{'lemma': 'Robert Downey Jr',
  'original_form': 'Robert Downey Jr',
  'tag': 'NPUU-N-'},
 {'lemma': 'Robert Downey Jr',
  'original_form': 'Robert Downey Jr',
  'tag': 'GNUS3S--'},
 {'lemma': 'top', 'original_form': 'has topped', 'tag': 'VI-S3PPA-N-N9'},
 {'id': 'ODENTITY_MAGAZINE',
  'lemma': 'Forbes',
  'original_form': 'Forbes',
  'tag': 'NP-S-N-'},
 {'lemma': 'magazine', 'original_form': 'magazine', 'tag': 'NC-S-N5'},
 {'lemma': 'magazine', 'original_form': 'Forbes magazine', 'tag': 'GN-S3---'},
 {'lemma': "'s", 'original_form': "'s", 'tag': 'WN-'},
 {'lemma': 'annual', 'original_form': 'annual', 'tag': 'AP-N5'},
 {'lemma': 'list', 'original_form': 'list', 'tag': 'NC-S-N5'},
 {'lemma': 'list', 'original_form': 'annual list', 'tag': 'GN-S3---'},
 {'id': 'ODENTITY_INDUSTRIAL_COMPANY',
  'lemma': 'John Deere',
  'original_form': 'John Deere',
  'tag': 'NP-S-N-'},
 {'lemma': 'John Deere', 'original_form': 'John Deere', 'tag': 'GN-S3Y--'},
 {'lemma': 'John Deere',
  'original_form': 'annual list John Deere',
  'tag': 'GN-S3---'},
 {'lemma': 'John Deere',
  'original_form': "Forbes magazine's annual list John Deere",
  'tag': 'GN-S3D--'},
 {'lemma': '*',
  'original_form': "Robert Downey Jr has topped Forbes magazine's annual list "
                   'John Deere',
  'tag': 'Z-----------'}]