BeautifulSoup 的解析器问题答案

【问题标题】：Parser Issue with BeautifulSoupBeautifulSoup 的解析器问题
【发布时间】：2018-06-08 12:40:14
【问题描述】：

我有一个循环遍历多个页面的脚本，它运行良好，直到它到达第 19 页。我收到错误消息：

if container.find(text="License: ").nextSibling.img:
AttributeError: 'NoneType' object has no attribute 'nextSibling'

该元素存在于第 19 页。我还包含了一个 else 语句，说明它何时不存在。我尝试使用requests 而不是urlopen 来查看这是否有所作为，甚至将解析器从html.parser 更改为html5lib 到lxml，但没有运气。我认为这可能是解析器问题，但我不确定解决方案可能是什么。这是我的脚本：

from urllib.request import urlopen
from bs4 import BeautifulSoup as soup
import json

base_url = "https://www.doabooks.org/"

data = []
n = 85
for i in range(1, n+1):
    if (i == 1):
    # handle first page
       response = urlopen(base_url)
    response = urlopen(base_url + "doab?func=browse&page=" + str(i) + "&queryField=A&uiLanguage=en")
    page_html = response.read()
    response.close()

    #html parsing
    page_soup = soup(page_html, "html5lib")

    #grabs info for each book
    containers = page_soup.findAll("div",{"class":"data"})

    for container in containers:
       item = {}
       item['type'] = "Open Access Book"
       item['title'] = container.span.text.strip()
       item['author'] =container.a.text
       item['link'] = "https://www.doabooks.org" + container.find('a', {'itemprop' : 'url'})['href']
       item['source'] = "Directory of Open Access Books"
       if container.div.find('a', {'itemprop' : 'about'}):
          item['subject'] = container.div.find('a', {'itemprop' : 'about'}).text.lstrip()
       else: 
          item['subject'] = ''
       item['base_url'] = "https://www.doabooks.org/"
       if container.find(text="License: ").nextSibling.img:
           item['license'] = container.find(text="License: ").nextSibling['href']
       else:
           item['license'] = container.find(text="License: ").nextSibling.text

       item['license_url'] = container.find(text="License: ").nextSibling['href']
       data.append(item) # add the item to the list

   with open("./json/doab-a.json", "w") as writeJSON:
      json.dump(data, writeJSON, ensure_ascii=False)

【问题讨论】：

标签： json python-3.x beautifulsoup

【解决方案1】：

https://www.doabooks.org/doab?func=browse&page=19&queryField=A&uiLanguage=en 上标题为“转换后”的书没有给出“许可证”，所以在你的代码中

container.find(text="License: ")

是 None 所以你不能得到 p>

.nextSibling.img

来自 NoneType 对象，因此它会引发异常。试试这样的：

if not container.find(string="License: "):
    item['license_url'] = item['license'] = "Not Specified"
else:
    if container.find(string="License: ").nextSibling.img:
        item['license'] = container.find(string="License: ").nextSibling['href']
    else:
        item['license'] = container.find(string="License: ").nextSibling.text
    item['license_url'] = container.find(string="License: ").nextSibling['href']

【讨论】：