Beautiful Soup 无法提取链接答案

【问题标题】：Beautiful Soup can't extract linksBeautiful Soup 无法提取链接
【发布时间】：2019-05-22 15:03:42
【问题描述】：

我正在尝试提取此网页的链接：https://search.cisco.com/search?query=iot

使用此代码我没有得到任何返回：

# Get Html Data from webpage
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html5lib')
# Retrieve all of the anchor tags    
tags = soup('a') for tag in tags:
    print(tag.get('href'))

我尝试了find_all() 方法，但遇到了同样的问题。

【问题讨论】：

请提供你得到的输出。
循环语句的格式是否有原因。我觉得这很奇怪。
@ottovon 我什么也没得到。循环语句很好。我只是不太干净地应对stackoverflow

标签： html python-3.x web-scraping beautifulsoup

【解决方案1】：

看起来像java脚本渲染到页面。你可以使用硒和美丽的汤来获取链接。

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get("https://search.cisco.com/search?query=iot&locale=enUS")
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()

for a in soup.find_all('a', href=True):
    print(a['href'])

输出：

https://onesearch.cloudapps.cisco.com/searchpage?queryFilter=iot
/login?query=iot&locale=enUS
/login?query=iot&locale=enUS
https://secure.opinionlab.com/ccc01/o.asp?id=pGuoWfLm&static=1&custom_var=undefined%7CS%7CenUS%7Ciot%7Cundefined%7CNA
https://www.cisco.com/c/en/us/support/index.html
//www.cisco.com/en/US/support/tsd_most_requested_tools.html
https://apps.cisco.com/WOC/WOConfigUI/pages/configset/configset.jsp?flow=nextgen&createNewConfigSet=Y
http://www.cisco-servicefinder.com/ServiceFinder.aspx
http://www.cisco-servicefinder.com/WarrantyFinder.aspx
//www.cisco.com/web/siteassets/sitemap/index.html    
https://www.cisco.com/c/dam/en/us/products/collateral/se/internet-of-things/at-a-glance-c45-731471.pdf?dtid=osscdc000283
https://www.cisco.com/c/en/us/solutions/internet-of-things/overview.html?dtid=osscdc000283
https://www.cisco.com/c/en/us/solutions/internet-of-things/iot-kinetic.html?dtid=osscdc000283
https://www.cisco.com/c/m/en_us/solutions/internet-of-things/iot-system.html?dtid=osscdc000283
https://learningnetworkstore.cisco.com/internet-of-things?dtid=osscdc000283
https://connectedfutures.cisco.com/tag/internet-of-things/?dtid=osscdc000283
https://blogs.cisco.com/internet-of-things?dtid=osscdc000283
https://learningnetwork.cisco.com/community/internet_of_things?dtid=osscdc000283
https://learningnetwork.cisco.com/community/learning_center/training-catalog/internet-of-things?dtid=osscdc000283
https://blogs.cisco.com/digital/internet-of-things-at-mwc?dtid=osscdc000283
https://cwr.cisco.com/
https://engage2demand.cisco.com/LP=4213?dtid=osscdc000283
https://engage2demand.cisco.com/LP=15823?dtid=osscdc000283
https://video.cisco.com/detail/video/4121788948001/internet-of-things:-empowering-the-enterprise?dtid=osscdc000283
https://video.cisco.com/detail/video/4121788948001/internet-of-things:-empowering-the-enterprise?dtid=osscdc000283
https://video.cisco.com/detail/video/3740968721001/protecting-the-internet-of-things?dtid=osscdc000283
https://video.cisco.com/detail/video/3740968721001/protecting-the-internet-of-things?dtid=osscdc000283
https://video.cisco.com/detail/video/4657296333001/the-internet-of-things:-the-vision-and-new-directions-ahead?dtid=osscdc000283
https://video.cisco.com/detail/video/4657296333001/the-internet-of-things:-the-vision-and-new-directions-ahead?dtid=osscdc000283
/search/videos?locale=enUS&query=iot
/search/videos?locale=enUS&query=iot
https://secure.opinionlab.com/ccc01/o.asp?id=pGuoWfLm&static=1&custom_var=undefined%7CS%7CenUS%7Ciot%7Cundefined%7CNA

【讨论】：

【解决方案2】：

你不需要硒。最好使用请求。该页面使用 API，因此来自该页面的请求

import requests

body = {"query":"iot","startIndex":0,"count":10,"searchType":"CISCO","tabName":"Cisco","debugScoreExplain":"false","facets":[],"localeStr":"enUS","advSearchFields":{"allwords":"","phrase":"","words":"","noOfWords":"","occurAt":""},"sortType":"RELEVANCY","isAdvanced":"false","dynamicRelevancyId":"","accessLevel":"","breakpoint":"XS","searchProfile":"","ui":"one","searchCat":"","searchMode":"text","callId":"j5JwndwQZZ","requestId":1558540148392,"bizCtxt":"","qnaTopic":[],"appName":"CDCSearhFE","social":"false"}
r = requests.post('https://search.cisco.com/api/search', json = body).json()

for item in r['items']:
    print(item['url'])

更改参数以获得更多结果等

【讨论】：

我还没有完全理解代码，但它可以工作。谢谢！
API 应该是您的首选。他们专门为您提供这些内容。基本上，您将查询发送到正文 "query":"iot" 。为了分批检索结果，您指定 startIndex，然后指定要返回的结果数作为 count 参数。
我怀疑有 API 文档告诉你结果的最大数量等。如果不四处寻找这个。然后，您可以循环更新正文以检索所有结果。
好的，非常感谢。我一定会深入研究这一点。真的有可能在不使用例如的情况下抓取整个 ~ 90 000 个搜索结果吗？ “飞溅”还是“硒”？您可能已经看到，有一个“加载更多”按钮可以动态加载更多搜索结果。
我会这么认为。探索每天的请求限制和每个查询的最大结果。该 API 专门用于此搜索查询目的。

【解决方案3】：

尝试按照documentation中给出的模板：

for link in soup.find_all('a'):
    print(link.get('href'))

【讨论】：