Beautifulsoup 检索 href 列表

Question

感谢关注！ 我正在尝试在搜索结果中检索产品的 href。 例如这个页面：

但是，当我缩小到产品图像类时，检索到的 href 是图像链接.... 任何人都可以解决这个问题吗？提前致谢！

url = 'http://www.homedepot.com/b/Husky/N-5yc1vZrd/Ntk-All/Ntt-chest%2Band%2Bcabinet?Ntx=mode+matchall&NCNI-5'
content = urllib2.urlopen(url).read()
content = preprocess_yelp_page(content) 
soup = BeautifulSoup(content)

content = soup.findAll('div',{'class':'content dynamic'})
draft = str(content)
soup = BeautifulSoup(draft)
items = soup.findAll('div',{'class':'cell_section1'})
draft = str(items)
soup = BeautifulSoup(draft)
content = soup.findAll('div',{'class':'product-image'})
draft = str(content)
soup = BeautifulSoup(draft)

Answer 1

您不需要一遍又一遍地使用BeautifulSoup 加载每个找到的标签的内容。

使用CSS selectors 获取所有产品链接（div 和class="product-image" 下的a 标签）

import urllib2
from bs4 import BeautifulSoup

url = 'http://www.homedepot.com/b/Husky/N-5yc1vZrd/Ntk-All/Ntt-chest%2Band%2Bcabinet?Ntx=mode+matchall&NCNI-5'
soup = BeautifulSoup(urllib2.urlopen(url))

for link in soup.select('div.product-image > a:nth-of-type(1)'):
    print link.get('href')

打印：

http://www.homedepot.com/p/Husky-41-in-16-Drawer-Tool-Chest-and-Cabinet-Set-HOTC4016B1QES/205080371
http://www.homedepot.com/p/Husky-26-in-6-Drawer-Chest-and-Cabinet-Combo-Black-C-296BF16/203420937
http://www.homedepot.com/p/Husky-52-in-18-Drawer-Tool-Chest-and-Cabinet-Set-Black-HOTC5218B1QES/204825971
http://www.homedepot.com/p/Husky-26-in-4-Drawer-All-Black-Tool-Cabinet-H4TR2R/204648170
...

div.product-image > a:nth-of-type(1) CSS 选择器将匹配 div 下的每个第一个 a 标记，类为 product-image。

要将链接保存到列表中，请使用列表推导：

links = [link.get('href') for link in soup.select('div.product-image > a:nth-of-type(1)')]