捕获多个相同标签的更好方法？

Question

我正在尝试创建一个抓取下载链接的刮板，我想使用正则表达式，但这对我来说将是一场噩梦，我找到了这个名为 BeautifulSoup 的库，我正在尝试捕获div class="article-content" 的子项中的 url，即<p> 标签，这个<h3> 是 url 的名称，我不想将所有 url 合并到一个列表中，而是使用字典，这是关键name(<h3>) 和 value 是 url 列表，这里的代码就够了。

import requests
from bs4 import BeautifulSoup

def scrape():
    resp = requests.get('https://www.animeout.xyz/love-live-nijigasaki-gakuen-school-idol-doukoukai-1080p-300mb720p-150mbepisode-1/')
    soup = BeautifulSoup(resp.text,'html.parser')
    contents = soup.find('div',class_='article-content')
    output = {}
    for tag in contents.children:
        if tag.name == 'h3':
            name = tag.text
            links = []
            for sibling in tag.next_siblings:
                if sibling.name == 'p':
                    for link in sibling.find_all('a',text='Direct Download'):
                        links.append(link.get('href'))
                if sibling.name == 'h3':
                    output.update({name:links})
                    break

到目前为止，我只设法捕获了 1 个密钥，有没有 Pythonic 方法可以做到这一点？

Answer 1

你可能想试试这个：

import json
import re

import requests
from bs4 import BeautifulSoup


def scrape(source_url):
    soup = BeautifulSoup(
        requests.get(source_url).text,
        'html.parser',
    )
    headers = [
        h.getText() for h in soup.find_all("h3") if "Direct" in h.getText()
    ]
    links = [
        anchor["href"] for anchor
        in soup.find_all(lambda t: t.name == "a" and "Direct" in t.text)
    ]
    return {
        header: [
            link for link in links
            if re.search(r"\d{3,4}p", header).group(0) in link
        ] for header in headers
    }


data = scrape("https://www.animeout.xyz/love-live-nijigasaki-gakuen-school-idol-doukoukai-1080p-300mb720p-150mbepisode-1/")

print(json.dumps(data, indent=2))

您只有一个键的原因是键 必须 是唯一的，但链接的名称不是。用一些独特的东西来改变它，例如索引号或带有分辨率的系列标题。

样本输出：

{
  "Love Live! Nijigasaki Gakuen School Idol Doukoukai (main) Direct Download Links (300MB \u2013 1080p)(Encoded)": [
    "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 01 [1080pp][1080pp][Erai-raws][RapidBot].mkv",
    "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 01 [v2][1080pp][1080pp][Erai-raws][RapidBot].mkv",
    "http://nimbus.animeout.com/series/00RAPIDBOT/Love Live Nijigasaki Gakuen School Idol Doukoukai/[AnimeOut] Love Live Nijigasaki Gakuen School Idol Doukoukai - 01 [1080pp][1080pp][Erai-raws][RapidBot].mkv",

and so on ...