如何从 a 下的 href 中提取所有 Urls，但它似乎总是给我一个错误

Question

category_tag = soup.find_all('div' , {'class': '_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8'})

category_tag 的输出：

<div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318158031">Action &amp; Adventure</a></div>,
 <div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318052031">Arts, Film &amp; Photography</a></div>,
 <div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318064031">Biographies, Diaries &amp; True Accounts</a></div>,
 <div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318068031">Business &amp; Economics</a></div>,
 <div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318073031">Children's &amp; Young Adult</a></div>,
 <div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318104031">Comics &amp; Mangas</a></div>,
 <div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318105031">Computing, Internet &amp; Digital Media</a></div>,
 <div class="_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8" role="treeitem"><a href="/gp/bestsellers/books/1318118031">Crafts, Home &amp; Lifestyle</a></div>,

现在的问题是，我无法从 '' 中提取 href。它一直显示错误。

我已经尝试过：

category_url_tag = category_tag.find('a')['href']

但它一直显示错误。

category_url = []
for tag in category_tag:
    category_url.append(tag.get('href'))
print(category_url)

这打印了一个包含None 的列表。

Answer 1

尝试选择更具体的元素，并在动态类上使用 id 和 tag 结构：

soup.select('#zg-left-col a')

或者更严格地说，只使用以特定模式开头的路径：

soup.select('#zg-left-col a[href^="/gp/bestsellers/books"]')

例子

import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(requests.get('https://www.amazon.in/gp/bestsellers/books/').text)


{'https://www.amazon.in'+a.get('href'):a.text for a in soup.select('#zg-left-col a[href^="/gp/bestsellers/books"]')}

输出

{'https://www.amazon.in/gp/bestsellers/books/1318158031': 'Action & Adventure',
 'https://www.amazon.in/gp/bestsellers/books/1318052031': 'Arts, Film & Photography',
 'https://www.amazon.in/gp/bestsellers/books/1318064031': 'Biographies, Diaries & True Accounts',
 'https://www.amazon.in/gp/bestsellers/books/1318068031': 'Business & Economics',
 'https://www.amazon.in/gp/bestsellers/books/1318073031': "Children's & Young Adult",
 'https://www.amazon.in/gp/bestsellers/books/1318104031': 'Comics & Mangas',
 'https://www.amazon.in/gp/bestsellers/books/1318105031': 'Computing, Internet & Digital Media',
 'https://www.amazon.in/gp/bestsellers/books/1318118031': 'Crafts, Home & Lifestyle',
 'https://www.amazon.in/gp/bestsellers/books/1318161031': 'Crime, Thriller & Mystery',
 'https://www.amazon.in/gp/bestsellers/books/22960344031': 'Engineering',...}

Answer 2

您正在遍历 div 和所有内容。你应该找到div的内部。

请检查以下代码。它应该给你预期的结果。

category_tag = soup.find_all('div' , {'class': '_p13n-zg-nav-tree-all_style_zg-browse-item__1rdKf _p13n-zg-nav-tree-all_style_zg-browse-height-large__1z5B8'})
categories = [(cat.find('a').text, cat.find('a')['href']) for cat in category_tag[1:]]