【问题标题】:Is there a way to convert CSV columns into hierarchical relationships?有没有办法将 CSV 列转换为层次关系?
【发布时间】:2020-03-08 16:32:07
【问题描述】:

我有一个包含 700 万条生物多样性记录的 csv,其中分类级别作为列。例如:

RecordID,kingdom,phylum,class,order,family,genus,species
1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris

我想在 D3 中创建可视化,但数据格式必须是网络,其中列的每个不同值都是特定值的前一列的子级。我需要从 csv 变成这样的东西:

{
  name: 'Animalia',
  children: [{
    name: 'Chordata',
    children: [{
      name: 'Mammalia',
      children: [{
        name: 'Primates',
        children: 'Hominidae'
      }, {
        name: 'Carnivora',
        children: 'Canidae'
      }]
    }]
  }]
}

我还没有想到如何在不使用一千个 for 循环的情况下做到这一点。有人对如何在 python 或 javascript 上创建这个网络有任何建议吗?

【问题讨论】:

  • 与您的问题无关,但就在我写完答案后,我注意到包含 Magnoliopsida 的门的 nan。那是什么nan? Phylum 是 Anthophyta,或者 Magnolia(它是旧的被子植物门)。

标签: javascript python d3.js data-visualization hierarchical-data


【解决方案1】:

基于代码生成库的替代选项 - convtools

from convtools import conversion as c
from convtools.contrib.tables import Table


table = Table.from_csv("tmp2.csv", header=True).drop("RecordID")

child = None
for level in reversed(table.columns):
    if child is None:
        child = c.iter(c.item(level)).as_type(list)
    else:
        child = c.group_by(c.item(level)).aggregate(
            {
                "name": c.item(level),
                "children": c.ReduceFuncs.Array(c.this()).pipe(child),
            }
        )

# this is where code generation happens
converter = child.gen_converter()

converter(table.into_iter_rows(dict))

输出以下内容:

[{'name': 'Animalia',
  'children': [{'name': 'Chordata',
    'children': [{'name': 'Mammalia',
      'children': [{'name': 'Primates',
        'children': [{'name': 'Hominidae',
          'children': [{'name': 'Homo', 'children': ['Homo sapiens']}]}]},
       {'name': 'Carnivora',
        'children': [{'name': 'Canidae',
          'children': [{'name': 'Canis', 'children': ['Canis']}]}]}]}]}]},
 {'name': 'Plantae',
  'children': [{'name': 'nan',
    'children': [{'name': 'Magnoliopsida',
      'children': [{'name': 'Brassicales',
        'children': [{'name': 'Brassicaceae',
          'children': [{'name': 'Arabidopsis',
            'children': ['Arabidopsis thaliana']}]}]},
       {'name': 'Fabales',
        'children': [{'name': 'Fabaceae',
          'children': [{'name': 'Phaseoulus',
            'children': ['Phaseolus vulgaris']}]}]}]}]}]}]

【讨论】:

    【解决方案2】:

    这是一个简单的 Python 解决方案。

    这个想法是有一个结构来快速找到层次结构中每个级别的子列表。 算法逐行读取输入,所以内存只花在树本身+一个小索引上,其大小取决于唯一王国、类等的数量

    import io
    import csv
    
    data = """RecordID,kingdom,phylum,class,order,family,genus,species
    1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
    2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
    3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
    4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris"""
    
    reader = csv.reader(io.StringIO(data))
    header = next(reader)
    
    result = []
    indexes = {i: {} for i in range(len(header))}
    
    for row in reader:
        children = result  # a pointer to current children list
        for level, item in enumerate(row[1:], start=1):
            if item not in indexes[level]:
                # add a new children if not exist
                indexes[level][item] = len(children)
                children.append({"name": item, "children": []})
    
            # shift pointer to the next level of the hierarchy
            idx = indexes[level][item]
            children = children[idx]["children"]
    
    print(result)
    

    输出:

    [{'name': 'Animalia',
      'children': [{'name': 'Chordata',
        'children': [{'name': 'Mammalia',
          'children': [{'name': 'Primates',
            'children': [{'name': 'Hominidae',
              'children': [{'name': 'Homo',
                'children': [{'name': 'Homo sapiens', 'children': []}]}]}]},
           {'name': 'Carnivora',
            'children': [{'name': 'Canidae',
              'children': [{'name': 'Canis',
                'children': [{'name': 'Canis', 'children': []}]}]}]}]}]}]},
     {'name': 'Plantae',
      'children': [{'name': 'nan',
        'children': [{'name': 'Magnoliopsida',
          'children': [{'name': 'Brassicales',
            'children': [{'name': 'Brassicaceae',
              'children': [{'name': 'Arabidopsis',
                'children': [{'name': 'Arabidopsis thaliana',
                  'children': []}]}]}]},
           {'name': 'Fabales',
            'children': [{'name': 'Fabaceae',
              'children': [{'name': 'Phaseoulus',
                'children': [{'name': 'Phaseolus vulgaris',
                  'children': []}]}]}]}]}]}]}]
    

    【讨论】:

      【解决方案3】:

      使用 python 和 python-benedict 库很容易完全满足您的需求(它在 Github 上是开源的,注意:我是作者):

      安装pip install python-benedict

      from benedict import benedict as bdict
      
      # data source can be a filepath or an url
      data_source = """
      RecordID,kingdom,phylum,class,order,family,genus,species
      1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
      2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
      3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
      4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris
      """
      data_input = bdict.from_csv(data_source)
      data_output = bdict()
      
      ancestors_hierarchy = ['kingdom', 'phylum', 'class', 'order', 'family', 'genus', 'species']
      for value in data_input['values']:
          data_output['.'.join([value[ancestor] for ancestor in ancestors_hierarchy])] = bdict()
      
      print(data_output.dump())
      # if this output is ok for your needs, you don't need the following code
      
      keypaths = sorted(data_output.keypaths(), key=lambda item: len(item.split('.')), reverse=True)
      
      data_output['children'] = []
      def transform_data(d, key, value):
          if isinstance(value, dict):
              value.update({ 'name':key, 'children':[] })
      data_output.traverse(transform_data)
      
      for keypath in keypaths:
          target_keypath = '.'.join(keypath.split('.')[:-1] + ['children'])
          data_output[target_keypath].append(data_output.pop(keypath))
      
      print(data_output.dump())
      

      第一个打印输出将是:

      {
          "Animalia": {
              "Chordata": {
                  "Mammalia": {
                      "Carnivora": {
                          "Canidae": {
                              "Canis": {
                                  "Canis": {}
                              }
                          }
                      },
                      "Primates": {
                          "Hominidae": {
                              "Homo": {
                                  "Homo sapiens": {}
                              }
                          }
                      }
                  }
              }
          },
          "Plantae": {
              "nan": {
                  "Magnoliopsida": {
                      "Brassicales": {
                          "Brassicaceae": {
                              "Arabidopsis": {
                                  "Arabidopsis thaliana": {}
                              }
                          }
                      },
                      "Fabales": {
                          "Fabaceae": {
                              "Phaseoulus": {
                                  "Phaseolus vulgaris": {}
                              }
                          }
                      }
                  }
              }
          }
      }
      

      第二个打印输出将是:

      {
          "children": [
              {
                  "name": "Animalia",
                  "children": [
                      {
                          "name": "Chordata",
                          "children": [
                              {
                                  "name": "Mammalia",
                                  "children": [
                                      {
                                          "name": "Carnivora",
                                          "children": [
                                              {
                                                  "name": "Canidae",
                                                  "children": [
                                                      {
                                                          "name": "Canis",
                                                          "children": [
                                                              {
                                                                  "name": "Canis",
                                                                  "children": []
                                                              }
                                                          ]
                                                      }
                                                  ]
                                              }
                                          ]
                                      },
                                      {
                                          "name": "Primates",
                                          "children": [
                                              {
                                                  "name": "Hominidae",
                                                  "children": [
                                                      {
                                                          "name": "Homo",
                                                          "children": [
                                                              {
                                                                  "name": "Homo sapiens",
                                                                  "children": []
                                                              }
                                                          ]
                                                      }
                                                  ]
                                              }
                                          ]
                                      }
                                  ]
                              }
                          ]
                      }
                  ]
              },
              {
                  "name": "Plantae",
                  "children": [
                      {
                          "name": "nan",
                          "children": [
                              {
                                  "name": "Magnoliopsida",
                                  "children": [
                                      {
                                          "name": "Brassicales",
                                          "children": [
                                              {
                                                  "name": "Brassicaceae",
                                                  "children": [
                                                      {
                                                          "name": "Arabidopsis",
                                                          "children": [
                                                              {
                                                                  "name": "Arabidopsis thaliana",
                                                                  "children": []
                                                              }
                                                          ]
                                                      }
                                                  ]
                                              }
                                          ]
                                      },
                                      {
                                          "name": "Fabales",
                                          "children": [
                                              {
                                                  "name": "Fabaceae",
                                                  "children": [
                                                      {
                                                          "name": "Phaseoulus",
                                                          "children": [
                                                              {
                                                                  "name": "Phaseolus vulgaris",
                                                                  "children": []
                                                              }
                                                          ]
                                                      }
                                                  ]
                                              }
                                          ]
                                      }
                                  ]
                              }
                          ]
                      }
                  ]
              }
          ]
      }
      

      【讨论】:

        【解决方案4】:

        其实@Charles Merriam 的解决方案非常优雅。

        如果你想让结果与问题相同,那么尝试如下。

        from io import StringIO
        import csv
        
        
        CSV_CONTENTS = """RecordID,kingdom,phylum,class,order,family,genus,species
        1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
        2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
        3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
        4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris
        """
        
        
        def recursive(dict_data):
            lst = []
            for key, val in dict_data.items():
                children = recursive(val)
                lst.append(dict(name=key, children=children))
            return lst
        
        
        def main():
            with StringIO() as io_f:
                io_f.write(CSV_CONTENTS)
                io_f.seek(0)
                io_f.readline()  # skip the column headers line of the file
                result_tree = {}
                for row_data in csv.reader(io_f):
                    cur_dict = result_tree  # cursor, back to root
                    for item in row_data[1:]:  # each item, skip the record number
                        if item not in cur_dict:
                            cur_dict[item] = {}  # create new dict
                            cur_dict = cur_dict[item]
                        else:
                            cur_dict = cur_dict[item]
        
            # change answer format
            result_list = []
            for cur_kingdom_name in result_tree:
                result_list.append(dict(name=cur_kingdom_name, children=recursive(result_tree[cur_kingdom_name])))
        
            # Optional
            import json
            from os import startfile
            output_file = 'result.json'
            with open(output_file, 'w') as f:
                json.dump(result_list, f)
            startfile(output_file)
        
        
        if __name__ == '__main__':
            main()
        
        

        【讨论】:

          【解决方案5】:

          为了创建您想要的精确嵌套对象,我们将混合使用纯 JavaScript 和名为 d3.stratify 的 D3 方法。但是,请记住,700 万行(请参阅下面的post scriptum)需要计算很多。

          值得一提的是,对于这个提议的解决方案,您必须在不同的数据数组中分隔王国(例如,使用Array.prototype.filter)。出现这种限制是因为我们需要一个根节点,并且在林奈分类法中,王国之间没有关系(除非您将 "Domain" 创建为最高等级,这将是所有真核生物的根,但是那么古细菌和细菌也会遇到同样的问题)。

          所以,假设您有这个只有一个王国的 CSV(我添加了更多行):

          RecordID,kingdom,phylum,class,order,family,genus,species
          1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
          2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis latrans
          3,Animalia,Chordata,Mammalia,Cetacea,Delphinidae,Tursiops,Tursiops truncatus
          1,Animalia,Chordata,Mammalia,Primates,Hominidae,Pan,Pan paniscus
          

          基于该 CSV,我们将在此处创建一个名为 tableOfRelationships 的数组,顾名思义,它具有等级之间的关系:

          const data = d3.csvParse(csv);
          
          const taxonomicRanks = data.columns.filter(d => d !== "RecordID");
          
          const tableOfRelationships = [];
          
          data.forEach(row => {
            taxonomicRanks.forEach((d, i) => {
              if (!tableOfRelationships.find(e => e.name === row[d])) tableOfRelationships.push({
                name: row[d],
                parent: row[taxonomicRanks[i - 1]] || null
              })
            })
          });
          

          对于上面的数据,这是tableOfRelationships

          +---------+----------------------+---------------+
          | (Index) |         name         |    parent     |
          +---------+----------------------+---------------+
          |       0 | "Animalia"           | null          |
          |       1 | "Chordata"           | "Animalia"    |
          |       2 | "Mammalia"           | "Chordata"    |
          |       3 | "Primates"           | "Mammalia"    |
          |       4 | "Hominidae"          | "Primates"    |
          |       5 | "Homo"               | "Hominidae"   |
          |       6 | "Homo sapiens"       | "Homo"        |
          |       7 | "Carnivora"          | "Mammalia"    |
          |       8 | "Canidae"            | "Carnivora"   |
          |       9 | "Canis"              | "Canidae"     |
          |      10 | "Canis latrans"      | "Canis"       |
          |      11 | "Cetacea"            | "Mammalia"    |
          |      12 | "Delphinidae"        | "Cetacea"     |
          |      13 | "Tursiops"           | "Delphinidae" |
          |      14 | "Tursiops truncatus" | "Tursiops"    |
          |      15 | "Pan"                | "Hominidae"   |
          |      16 | "Pan paniscus"       | "Pan"         |
          +---------+----------------------+---------------+
          

          看看null 作为Animalia 的父级:这就是为什么我告诉你你需要按Kingdoms 分隔数据集,整个表中只能有一个null 值。

          最后,基于该表,我们使用d3.stratify() 创建层次结构:

          const stratify = d3.stratify()
              .id(function(d) { return d.name; })
              .parentId(function(d) { return d.parent; });
          
          const hierarchicalData = stratify(tableOfRelationships);
          

          这里是演示。打开浏览器的控制台(sn-p 的控制台不太适合此任务)并检查对象的多个级别 (children):

          const csv = `RecordID,kingdom,phylum,class,order,family,genus,species
          1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
          2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis latrans
          3,Animalia,Chordata,Mammalia,Cetacea,Delphinidae,Tursiops,Tursiops truncatus
          1,Animalia,Chordata,Mammalia,Primates,Hominidae,Pan,Pan paniscus`;
          
          const data = d3.csvParse(csv);
          
          const taxonomicRanks = data.columns.filter(d => d !== "RecordID");
          
          const tableOfRelationships = [];
          
          data.forEach(row => {
            taxonomicRanks.forEach((d, i) => {
              if (!tableOfRelationships.find(e => e.name === row[d])) tableOfRelationships.push({
                name: row[d],
                parent: row[taxonomicRanks[i - 1]] || null
              })
            })
          });
          
          const stratify = d3.stratify()
            .id(function(d) {
              return d.name;
            })
            .parentId(function(d) {
              return d.parent;
            });
          
          const hierarchicalData = stratify(tableOfRelationships);
          
          console.log(hierarchicalData);
          <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>

          PS:我不知道你会创建什么样的数据可视化,但你真的应该避免分类等级。整个林奈分类法已经过时,我们不再使用等级:因为系统发育系统学是在 60 年代中期开发的,我们只使用分类群,没有任何分类等级(这里是进化生物学老师)。另外,我对这 700 万行很好奇,因为我们已经描述了超过 100 万个物种!

          【讨论】:

          • .@gerardo 感谢您的回答,我会看看它是否适用于 7M 行的样本。该数据库包含许多物种的重复行。所以想法是显示某个分类等级有多少条记录。这个想法是创建类似于Mike Bostock's Zoomable Icicle Tree 的东西。
          【解决方案6】:

          一个有趣的挑战。试试这个 javascript 代码。为简单起见,我使用 Lodash 的套件。

          import { set } from 'lodash'
          
          const csvString = `RecordID,kingdom,phylum,class,order,family,genus,species
              1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
              2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
              3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
              4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris`
          
          // First create a quick lookup map
          const result = csvString
            .split('\n') // Split for Rows
            .slice(1) // Remove headers
            .reduce((acc, row) => {
              const path = row
                .split(',') // Split for columns
                .filter(item => item !== 'nan') // OPTIONAL: Filter 'nan'
                .slice(1) // Remove record id
              const species = path.pop() // Pull out species (last entry)
              set(acc, path, species)
              return acc
            }, {})
          
          console.log(JSON.stringify(result, null, 2))
          
          // Then convert to the name-children structure by recursively calling this function
          const convert = (obj) => {
            // If we're at the end of our chain, end the chain (children is empty)
            if (typeof obj === 'string') {
              return [{
                name: obj,
                children: [],
              }]
            }
            // Else loop through each entry and add them as children
            return Object.entries(obj)
              .reduce((acc, [key, value]) => acc.concat({
                name: key,
                children: convert(value), // Recursive call
              }), [])
          }
          
          const result2 = convert(result)
          
          console.log(JSON.stringify(result2, null, 2))
          

          这会产生与您想要的结果(相似)。

          [
            {
              "name": "Animalia",
              "children": [
                {
                  "name": "Chordata",
                  "children": [
                    {
                      "name": "Mammalia",
                      "children": [
                        {
                          "name": "Primates",
                          "children": [
                            {
                              "name": "Hominidae",
                              "children": [
                                {
                                  "name": "Homo",
                                  "children": [
                                    {
                                      "name": "Homo sapiens",
                                      "children": []
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        },
                        {
                          "name": "Carnivora",
                          "children": [
                            {
                              "name": "Canidae",
                              "children": [
                                {
                                  "name": "Canis",
                                  "children": [
                                    {
                                      "name": "Canis",
                                      "children": []
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "name": "Plantae",
              "children": [
                {
                  "name": "Magnoliopsida",
                  "children": [
                    {
                      "name": "Brassicales",
                      "children": [
                        {
                          "name": "Brassicaceae",
                          "children": [
                            {
                              "name": "Arabidopsis",
                              "children": [
                                {
                                  "name": "Arabidopsis thaliana",
                                  "children": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "name": "Fabales",
                      "children": [
                        {
                          "name": "Fabaceae",
                          "children": [
                            {
                              "name": "Phaseoulus",
                              "children": [
                                {
                                  "name": "Phaseolus vulgaris",
                                  "children": []
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
          

          【讨论】:

            【解决方案7】:

            这看起来很简单,所以也许我不明白你的问题。

            您想要的数据结构是一组嵌套的字典、键/值对。您的顶级王国字典为您的每个王国都有一个键,其值是门字典。一个门字典(对于一个王国)对每个门名称都有一个键,每个键都有一个类字典的值,依此类推。

            为了简化编码,您的属字典将为每个物种都有一个键,但该物种的值将是空字典。

            这应该是你想要的;不需要奇怪的库。

            import csv
            
            def read_data(filename):
                tree = {}
                with open(filename) as f:
                    f.readline()  # skip the column headers line of the file
                    for animal_cols in csv.reader(f):
                        spot = tree
                        for name in animal_cols[1:]:  # each name, skipping the record number
                            if name in spot:  # The parent is already in the tree
                                spot = spot[name]  
                            else:
                                spot[name] = {}  # creates a new entry in the tree
                                spot = spot[name]
                return tree
            

            为了测试它,我使用了您的数据和标准库中的pprint

            from pprint import pprint
            pprint(read_data('data.txt'))
            

            得到

            {'Animalia': {'Chordata': {'Mammalia': {'Carnivora': {'Canidae': {'Canis': {'Canis': {}}}},
                                                    'Primates': {'Hominidae': {'Homo': {'Homo sapiens': {}}}}}}},
             'Plantae': {'nan': {'Magnoliopsida': {'Brassicales': {'Brassicaceae': {'Arabidopsis': {'Arabidopsis thaliana': {}}}},
                                                   'Fabales': {'Fabaceae': {'Phaseoulus': {'Phaseolus vulgaris': {}}}}}}}}
            

            再次阅读您的问题,您可能需要一个大表(“来自更一般的组的链接”、“链接到更具体的组”)。也就是说,“Animalia”链接到“Animalia:Chordata”,“Animalia:Chordata”链接到“Animalia:Chordata:Mammalia”等。不幸的是,数据中的“nan”意味着每个链接都需要全名。如果( parent, child) 对是你想要的,这样走树:

            def walk_children(tree, parent=''):
                for child in tree.keys():
                    full_name = parent + ':' + child
                    yield (parent, full_name)
                    yield from walk_children(tree[child], full_name)
            
            tree = read_data('data.txt')
            for (parent, child) in walk_children(tree):
                print(f'parent="{parent}" child="{child}"')
            

            给予:

            parent="" child=":Animalia"
            parent=":Animalia" child=":Animalia:Chordata"
            parent=":Animalia:Chordata" child=":Animalia:Chordata:Mammalia"
            parent=":Animalia:Chordata:Mammalia" child=":Animalia:Chordata:Mammalia:Primates"
            parent=":Animalia:Chordata:Mammalia:Primates" child=":Animalia:Chordata:Mammalia:Primates:Hominidae"
            parent=":Animalia:Chordata:Mammalia:Primates:Hominidae" child=":Animalia:Chordata:Mammalia:Primates:Hominidae:Homo"
            parent=":Animalia:Chordata:Mammalia:Primates:Hominidae:Homo" child=":Animalia:Chordata:Mammalia:Primates:Hominidae:Homo:Homo sapiens"
            parent=":Animalia:Chordata:Mammalia" child=":Animalia:Chordata:Mammalia:Carnivora"
            parent=":Animalia:Chordata:Mammalia:Carnivora" child=":Animalia:Chordata:Mammalia:Carnivora:Canidae"
            parent=":Animalia:Chordata:Mammalia:Carnivora:Canidae" child=":Animalia:Chordata:Mammalia:Carnivora:Canidae:Canis"
            parent=":Animalia:Chordata:Mammalia:Carnivora:Canidae:Canis" child=":Animalia:Chordata:Mammalia:Carnivora:Canidae:Canis:Canis"
            parent="" child=":Plantae"
            parent=":Plantae" child=":Plantae:nan"
            parent=":Plantae:nan" child=":Plantae:nan:Magnoliopsida"
            parent=":Plantae:nan:Magnoliopsida" child=":Plantae:nan:Magnoliopsida:Brassicales"
            parent=":Plantae:nan:Magnoliopsida:Brassicales" child=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae"
            parent=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae" child=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae:Arabidopsis"
            parent=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae:Arabidopsis" child=":Plantae:nan:Magnoliopsida:Brassicales:Brassicaceae:Arabidopsis:Arabidopsis thaliana"
            parent=":Plantae:nan:Magnoliopsida" child=":Plantae:nan:Magnoliopsida:Fabales"
            parent=":Plantae:nan:Magnoliopsida:Fabales" child=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae"
            parent=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae" child=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae:Phaseoulus"
            parent=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae:Phaseoulus" child=":Plantae:nan:Magnoliopsida:Fabales:Fabaceae:Phaseoulus:Phaseolus vulgaris"
            

            【讨论】:

            • 这不会按照问题的要求返回带有namechildren 的嵌套字典。
            • 不,它没有。要求的是“类似这样的东西”;我认为这是试图找到想法数据结构。一个人可以通过走树来构建一个自定义结构,这是一个四行练习。
            【解决方案8】:

            将数据转换为层次结构的最简单方法可能是利用 D3 的内置 nesting 运算符 d3.nest()

            嵌套允许将数组中的元素分组为层次树结构;

            通过nest.key() 注册关键功能,您可以轻松指定层次结构。就像 Gerardo 在他的 answer 中阐述的一样,您可以在解析 CSV 后使用数据数组上公开的 .columns 属性来自动生成这些关键函数。整个代码归结为以下几行:

            const nester = d3.nest();                             // Create a nest operator
            const [, ...taxonomicRanks] = data.columns;           // Get rid of the RecordID property
            taxonomicRanks.forEach(r => nester.key(d => d[r]));   // Register key functions
            const nest = nester.entries(data);                    // Calculate hierarchy
            

            但是请注意,生成的层次结构与您问题中请求的结构并不完全相似,因为对象是 { key, values } 而不是 { name, children };顺便说一句,这同样适用于 Gerardo 的回答。不过,这对两个答案都没有什么坏处,因为通过指定 children accessor 函数,d3.hierarchy() 会导致结果拥塞:

            d3.hierarchy(nest, d => d.values)   // Second argument is the children accessor
            

            以下演示将所有部分放在一起:

            const csv = `RecordID,kingdom,phylum,class,order,family,genus,species
            1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
            2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis latrans
            3,Animalia,Chordata,Mammalia,Cetacea,Delphinidae,Tursiops,Tursiops truncatus
            1,Animalia,Chordata,Mammalia,Primates,Hominidae,Pan,Pan paniscus`;
            
            const data = d3.csvParse(csv);
            
            const nester = d3.nest();
            const [, ...taxonomicRanks] = data.columns;
            taxonomicRanks.forEach(r => nester.key(d => d[r]));
            const nest = nester.entries(data);
            
            console.log(nest);
            
            const hierarchy = d3.hierarchy(nest, d => d.values);
            
            console.log(hierarchy);
            <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.12.0/d3.js"></script>

            您可能还想查看d3.nest() key and values conversion to name and children,以防您觉得有必要准确了解您发布的结构。

            【讨论】:

            • 在它持续的时候享受d3.nest:它很快就会被弃用。
            • @GerardoFurtado 这是我自己的第一个想法。但是,我找不到任何支持这个假设的参考资料。我以为我已经阅读了有关它的删除信息,甚至惊讶地发现它仍然包含在捆绑包中。 d3-collection 已存档,但没有弃用说明。你有关于这件事的可靠信息吗?
            • 这是针对 v6 的,请看here。看看 "d3-collection [Removed!]".
            • @GerardoFurtado 不,这不是我想到的参考。不过,遗憾的是,它回答了我的问题。
            【解决方案9】:

            var log = console.log;
            var data = `
            1,Animalia,Chordata,Mammalia,Primates,Hominidae,Homo,Homo sapiens
            2,Animalia,Chordata,Mammalia,Carnivora,Canidae,Canis,Canis
            3,Plantae,nan,Magnoliopsida,Brassicales,Brassicaceae,Arabidopsis,Arabidopsis thaliana
            4,Plantae,nan,Magnoliopsida,Fabales,Fabaceae,Phaseoulus,Phaseolus vulgaris`;
            //make array of rows with array of values
            data = data.split("\n").map(v=>v.split(","));
            //init tree
            var tree = {};
            data.forEach(row=>{
                //set current = root of tree for every row
                var cur = tree; 
                var id = false;
                row.forEach((value,i)=>{
                    if (i == 0) {
                        //set id and skip value
                        id = value;
                        return;
                    }
                    //If branch not exists create. 
                    //If last value - write id
                    if (!cur[value]) cur[value] = (i == row.length - 1) ? id : {};
                    //Move link down on hierarhy
                    cur = cur[value];
                });
            }); 
            log("Tree:");
            log(JSON.stringify(tree, null, "  "));
            
            //Now you have hierarhy in tree and can do anything with it.
            var toStruct = function(obj) {
                let ret = [];
                for (let key in obj) {
                    let child = obj[key];
                    let rec = {};
                    rec.name = key;
                    if (typeof child == "object") rec.children = toStruct(child);
                    ret.push(rec);
                }
                return ret;
            }
            var struct = toStruct(tree);
            console.log("Struct:");
            console.log(struct);

            【讨论】:

              【解决方案10】:

              在 Python 中,对树进行编码的一种方法是使用 dict,其中的键代表节点,关联的值是节点的父节点:

              {'Homo sapiens': 'Homo',
               'Canis': 'Canidae',
               'Arabidopsis thaliana': 'Arabidopsis',
               'Phaseolus vulgaris': 'Phaseoulus',
               'Homo': 'Hominidae',
               'Arabidopsis': 'Brassicaceae',
               'Phaseoulus': 'Fabaceae',
               'Hominidae': 'Primates',
               'Canidae': 'Carnivora',
               'Brassicaceae': 'Brassicales',
               'Fabaceae': 'Fabales',
               'Primates': 'Mammalia',
               'Carnivora': 'Mammalia',
               'Brassicales': 'Magnoliopsida',
               'Fabales': 'Magnoliopsida',
               'Mammalia': 'Chordata',
               'Magnoliopsida': 'nan',
               'Chordata': 'Animalia',
               'nan': 'Plantae',
               'Animalia': None,
               'Plantae': None}
              

              这样做的一个好处是您可以确保节点是唯一的,因为dicts 不能有重复的键。

              如果您想编码一个更通用的有向图(即节点可以有多个父节点),您可以使用列表作为值并使用代表子节点(或父节点,我想):

              {'Homo': ['Homo sapiens', 'ManBearPig'],
              'Ursus': ['Ursus arctos', 'ManBearPig'],
              'Sus': ['ManBearPig']}
              

              你可以在 JS 中对 Objects 做类似的事情,如果需要的话,用 Arrays 代替列表。

              这是我用来创建上面第一个字典的 Python 代码:

              import csv
              
              ROWS = []
              # Load file: tbl.csv
              with open('tbl.csv', 'r') as in_file:
                  csvreader = csv.reader(in_file)
              
                  # Ignore leading row numbers
                  ROWS = [row[1:] for row in csvreader]
                  # Drop header row
                  del ROWS[0]
              
              # Build dict
              mytree = {row[i]: row[i-1] for row in ROWS for i in range(len(row)-1, 0, -1)}
              # Add top-level nodes
              mytree = {**mytree, **{row[0]: None for row in ROWS}}
              

              【讨论】:

                猜你喜欢
                • 1970-01-01
                • 2019-09-15
                • 1970-01-01
                • 1970-01-01
                • 2020-01-10
                • 2017-07-15
                • 2019-10-03
                • 1970-01-01
                • 2019-03-31
                相关资源
                最近更新 更多