BeautifulSoup 4：AttributeError：NoneType 没有属性 find_next

Question

项目：wordpress-plugins 的元数据列表： - 大约有 50 个插件感兴趣！但挑战是：我想获取所有现有插件的元数据。我随后想要在获取后过滤掉的是 - 那些具有最新时间戳的插件 - 最近更新（最近）。这一切都是为了 aobut acutality...所以开始的基本 URL 是这样的：

url = "https://wordpress.org/plugins/browse/popular/

目标：我想获取我们在流行插件的前 50 页上找到的插件的所有元数据......例如......：

https://wordpress.org/plugins/wp-job-manager
https://wordpress.org/plugins/ninja-forms
https://wordpress.org/plugins/participants-database ....and so on and so forth.

我们开始：

import requests
from bs4 import BeautifulSoup
from concurrent.futures.thread import ThreadPoolExecutor

url = "https://wordpress.org/plugins/browse/popular/{}"


def main(url, num):
    with requests.Session() as req:
        print(f"Collecting Page# {num}")
        r = req.get(url.format(num))
        soup = BeautifulSoup(r.content, 'html.parser')
        link = [item.get("href")
                for item in soup.findAll("a", rel="bookmark")]
        return set(link)


with ThreadPoolExecutor(max_workers=20) as executor:
    futures = [executor.submit(main, url, num)
               for num in [""]+[f"page/{x}/" for x in range(2, 50)]]

allin = []
for future in futures:
    allin.extend(future.result())


def parser(url):
    with requests.Session() as req:
        print(f"Extracting {url}")
        r = req.get(url)
        soup = BeautifulSoup(r.content, 'html.parser')
        target = [item.get_text(strip=True, separator=" ") for item in soup.find(
            "h3", class_="screen-reader-text").find_next("ul").findAll("li")[:8]]
        head = [soup.find("h1", class_="plugin-title").text]
        new = [x for x in target if x.startswith(
            ("V", "Las", "Ac", "W", "T", "P"))]
        return head + new


with ThreadPoolExecutor(max_workers=50) as executor1:
    futures1 = [executor1.submit(parser, url) for url in allin]

for future in futures1:
    print(future.result())

这样运行 - 但会返回一些错误..（见下文）

Extracting https://wordpress.org/plugins/use-google-libraries/
Extracting https://wordpress.org/plugins/blocksy-companion/
Extracting https://wordpress.org/plugins/cherry-sidebars/
Extracting https://wordpress.org/plugins/accesspress-social-share/Extracting https://wordpress.org/plugins/goodbye-captcha/
Extracting https://wordpress.org/plugins/wp-whatsapp/

这里是错误的追溯：

Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
Traceback (most recent call last):

  File "C:\Users\rob\.spyder-py3\dev\untitled0.py", line 51, in <module>
    print(future.result())

  File "C:\Users\rob\devel\IDE\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()

  File "C:\Users\rob\devel\IDE\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception

  File "C:\Users\rob\devel\IDE\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)

  File "C:\Users\rob\.spyder-py3\dev\untitled0.py", line 39, in parser
    target = [item.get_text(strip=True, separator=" ") for item in soup.find(

AttributeError: 'NoneType' object has no attribute 'find_next'

更新：如上所述，我得到了这个 AttributeError，它说 NoneType 没有属性 find_next。下面是产生令人讨厌的问题的行。

target = [item.get_text(strip=True, separator=" ") for item in soup.find("h3", class_="screen-reader-text").find_next("ul").findAll("li")]

具体来说，问题出在 soup.find() 方法中，该方法可以返回具有 .find_next() 方法（即属性）的 Tag（当它找到某些东西时）或 None（当它没有时）找到任何东西），但没有。我们可以尝试将整个调用提取到它自己的变量中，然后我们可以对其进行测试。

tag = soup.find("h3", class_="screen-reader-text")
target = []
if tag:
    lis = tag.find_next("ul").findAll("li")
    target = [item.get_text(strip=True, separator=" ") for item in lis[:8]]

顺便说一句;我们可以使用 CSS 选择器来运行它：

target = [item.get_text(strip=True, separator=" ") for item in soup.select("h3.screen-reader-text + ul li")[:8]]

这会得到“ul 下的所有 li，它与屏幕阅读器文本类紧邻 h3”。如果我们希望 li 直接位于 ul 之下（通常无论如何都是这样，但其他元素并非总是如此），我们可以使用 ul > li 代替（> 表示“直接子元素”）。

注意：最好的办法是将所有结果转储到 csv 文件中，或者 - 在屏幕上打印出来。

期待您的来信

Answer 1

该页面组织得很好，因此抓取它应该非常简单。您需要做的就是获取插件卡，然后简单地提取必要的部分。

这是我的看法。

import time

import pandas as pd
import requests
from bs4 import BeautifulSoup

main_url = "https://wordpress.org/plugins/browse/popular"
headers = [
    "Title", "Rating", "Rating Count", "Excerpt", "URL",
    "Author", "Active installs", "Tested with", "Last Updated",
]


def wait_a_bit(wait_for: float = 1.5):
    time.sleep(wait_for)


def parse_plugin_card(card) -> list:
    title = card.select_one("h3").getText()
    rating = card.select_one(
        ".plugin-rating .wporg-ratings"
    )["data-rating"]
    rating_count = card.select_one(
        ".plugin-rating .rating-count a"
    ).getText().replace(" total ratings", "")
    excerpt = card.select_one(
        ".plugin-card .entry-excerpt p"
    ).getText()
    plugin_author = card.select_one(
        ".plugin-card footer span.plugin-author"
    ).getText(strip=True)
    active_installs = card.select_one(
        ".plugin-card footer span.active-installs"
    ).getText(strip=True)
    tested_with = card.select_one(
        ".plugin-card footer span.tested-with"
    ).getText(strip=True)
    last_updated = card.select_one(
        ".plugin-card footer span.last-updated"
    ).getText(strip=True)
    plugin_url = card.select_one(
        ".plugin-card .entry-title a"
    )["href"]
    return [
        title, rating, rating_count, excerpt, plugin_url,
        plugin_author, active_installs, tested_with, last_updated,
    ]


with requests.Session() as connection:
    pages = (
        BeautifulSoup(
            connection.get(main_url).text,
            "lxml",
        ).select(".pagination .nav-links .page-numbers")
    )[-2].getText(strip=True)

    all_cards = []
    for page in range(1, int(pages) + 1):
        print(f"Scraping page {page} out of {pages}...")
        # deal with the first page
        page_link = f"{main_url}" if page == 1 else f"{main_url}/page/{page}"
        plugin_cards = BeautifulSoup(
            connection.get(page_link).text,
            "lxml",
        ).select(".plugin-card")
        for plugin_card in plugin_cards:
            all_cards.append(parse_plugin_card(plugin_card))
    wait_a_bit()

df = pd.DataFrame(all_cards, columns=headers)
df.to_csv("all_plugins.csv", index=False)

它抓取所有页面（目前有 49 个）并将所有内容转储到一个 .csv 文件中，其中包含 980 行（截至目前），如下所示：

你甚至不必运行代码，整个转储是here。

Answer 2

Baduker 的解决方案很棒，只是想补充一下。

我们可以稍微修改插件卡的解析，因为有一个 api 可以返回所有这些数据。仍然需要少量处理（即，为作者提取内容，我相信评分存储在 100 分中（因此 82 的评分实际上是 82/100*5 = 4.1 ->“4 星”），并且诸如此类。

但我想我会分享。

import time

import pandas as pd
import requests
from bs4 import BeautifulSoup

main_url = "https://wordpress.org/plugins/browse/popular"


def wait_a_bit(wait_for: float = 1.5):
    time.sleep(wait_for)


# MODIFICATION MADE HERE
def parse_plugin_card(card):
    plugin_slug = card.select_one('a')['href'].split('/')[-2]
    url = 'https://api.wordpress.org/plugins/info/1.0/%s.json' %plugin_slug
    jsonData = requests.get(url).json()
    sections = jsonData.pop('sections')
    for k, v in sections.items():
        sections[k] = BeautifulSoup(v).text
    jsonData.update(sections)
    return jsonData


with requests.Session() as connection:
    pages = (
        BeautifulSoup(
            connection.get(main_url).text,
            "lxml",
        ).select(".pagination .nav-links .page-numbers")
    )[-2].getText(strip=True)

    all_cards = []
    for page in range(1, int(pages) + 1):
        print(f"Scraping page {page} out of {pages}...")
        # deal with the first page
        page_link = f"{main_url}" if page == 1 else f"{main_url}/page/{page}"
        plugin_cards = BeautifulSoup(
            connection.get(page_link).text,
            "lxml",
        ).select(".plugin-card")
        for plugin_card in plugin_cards:
            all_cards.append(parse_plugin_card(plugin_card))
    wait_a_bit()

df = pd.DataFrame(all_cards)
df.to_csv("all_plugins.csv", index=False)

这里只是一个展示列的示例：