Beautiful Soup：如何判断 html 的递归解析何时嵌套答案

【问题标题】：Beautiful Soup: how to tell when recursive parsing of html is nestedBeautiful Soup：如何判断 html 的递归解析何时嵌套
【发布时间】：2017-03-09 18:46:17
【问题描述】：

我有一些我正在使用 BeautifulSoup 解析的 html。我正在使用以下代码从 dom 中获取所有类别：

def dataList(element):
    categoryList = []
    try:
        for ul in categorySoup('ul', recursive=False):
            for li in ul('li', recursive=True):
                categoryList.append(li.a.contents)


        return categoryList
    except:
        return ['broken!']

我遇到的问题是这段代码我不知道什么嵌套在什么里面。有没有一种方法可以在我进出 li/ul 的任何时候向 categoryList 添加附加项目，这样我就可以看到数据的层次结构是什么？ html 的基本/简化示例如下：

<ul>
  <li>category1
    <ul>
      <li>subcategory1
        <ul>
          <li>subsubcategory1</li>
          <li>subsubcategory2</li>
        </ul>
      <li>subcategory2</li>
     </ul>
   </li>
  <li>categorya
    <ul>
      <li>subcategorya</li>
      <li>subcategoryb
        <ul>
          <li>subsubcategorya</li>
        </ul>
      </li>
     </ul>
  </li>
</ul>

我给出的一个实际例子是这样的：

<li><b><a href="/getcats/fullcategorytree.php?RootID=20081#20081" name="20081">Antiques</a></b> #20081<ul>\n<li><b><a href="/products.php?extra=Antiquities&amp;cat=37903&amp;keywords=" rel="nofollow">Antiquities</a> #37903</b>\n<ul>\n<li><a class="elink" href="/products.php?extra=Byzantine&amp;cat=162922&amp;keywords=" rel="nofollow">Byzantine</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162922 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Celtic&amp;cat=162923&amp;keywords=" rel="nofollow">Celtic</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162923 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Egyptian&amp;cat=37905&amp;keywords=" rel="nofollow">Egyptian</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #37905 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Far+Eastern&amp;cat=162916&amp;keywords=" rel="nofollow">Far Eastern</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162916 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Greek&amp;cat=37906&amp;keywords=" rel="nofollow">Greek</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #37906 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Holy+Land&amp;cat=162917&amp;keywords=" rel="nofollow">Holy Land</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162917 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Islamic&amp;cat=162918&amp;keywords=" rel="nofollow">Islamic</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162918 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Near+Eastern&amp;cat=91101&amp;keywords=" rel="nofollow">Near Eastern</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #91101 (leaf)</img></li>\n<li><a class="elink" href="/products.php?extra=Neolithic+%26+Paleolithic&amp;cat=66834&amp;keywords=" rel="nofollow">Neolithic &amp; Paleolithic</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #66834 (leaf)</img></li>

不过，它们可以比 2 或 3 层更深。这是我们试图解析的实际来源：http://www.isoldwhat.com/getcats/fullcategorytree.php

【问题讨论】：

标签： python json parsing beautifulsoup html-parsing

【解决方案1】：

您可以执行递归函数来搜索（并存储在 dict 中）列表并将当前 ul 节点附加到最后找到的 li 中。

例子：

import requests
from bs4 import BeautifulSoup

example = """
<ul><li><b><a href="/getcats/fullcategorytree.php?RootID=20081#20081" name="20081">Antiques</a></b> #20081
<ul>\n
    <li><b><a href="/products.php?extra=Antiquities&amp;cat=37903&amp;keywords=" rel="nofollow">Antiquities</a> #37903</b>\n
        <ul>\n
            <li><a class="elink" href="/products.php?extra=Byzantine&amp;cat=162922&amp;keywords=" rel="nofollow">Byzantine</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162922 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Celtic&amp;cat=162923&amp;keywords=" rel="nofollow">Celtic</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162923 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Egyptian&amp;cat=37905&amp;keywords=" rel="nofollow">Egyptian</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #37905 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Far+Eastern&amp;cat=162916&amp;keywords=" rel="nofollow">Far Eastern</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162916 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Greek&amp;cat=37906&amp;keywords=" rel="nofollow">Greek</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #37906 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Holy+Land&amp;cat=162917&amp;keywords=" rel="nofollow">Holy Land</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162917 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Islamic&amp;cat=162918&amp;keywords=" rel="nofollow">Islamic</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #162918 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Near+Eastern&amp;cat=91101&amp;keywords=" rel="nofollow">Near Eastern</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #91101 (leaf)</img>
            </li>\n
            <li><a class="elink" href="/products.php?extra=Neolithic+%26+Paleolithic&amp;cat=66834&amp;keywords=" rel="nofollow">Neolithic &amp; Paleolithic</a>\xa0 <img alt="Go to eBay" height="9" src="/images/graphics/smallebay.gif" style="display:inline;" width="16"> #66834 (leaf)</img>
            </li></ul>"""

def Tree(ul):
    result = {}
    for li in ul.find_all("li", recursive=False):
        key = next(li.stripped_strings)
        HasUL = li.find("ul")
        #If you want only the node text
        result[key] = None if HasUL is None else Tree(HasUL)
        #If the li have a `a` tag and you want the url
        #result["%s - %s" % (key, li.find("a")["href"])] = None if HasUL is None else Tree(HasUL)
    return result

soup = BeautifulSoup(example, "html.parser")

print (Tree(soup.find("ul")))

第二种方法输出：

{'Antiques - /getcats/fullcategorytree.php?RootID=20081#20081': {'Antiquities - /products.php?extra=Antiquities&cat=37903&keywords=': {'Byzantine - /products.php?extra=Byzantine&cat=162922&keywords=': None, 'Celtic - /products.php?extra=Celtic&cat=162923&keywords=': None, 'Egyptian - /products.php?extra=Egyptian&cat=37905&keywords=': None, 'Far Eastern - /products.php?extra=Far+Eastern&cat=162916&keywords=': None, 'Greek - /products.php?extra=Greek&cat=37906&keywords=': None, 'Holy Land - /products.php?extra=Holy+Land&cat=162917&keywords=': None, 'Islamic - /products.php?extra=Islamic&cat=162918&keywords=': None, 'Near Eastern - /products.php?extra=Near+Eastern&cat=91101&keywords=': None, 'Neolithic & Paleolithic - /products.php?extra=Neolithic+%26+Paleolithic&cat=66834&keywords=': None}}}

【讨论】：