【发布时间】：2020-12-02 14:19:59
【问题描述】：
我对 python 很陌生，对网络抓取也很陌生 - 目前正在学习 Al Sweigart 的书Automate the Boring Stuff With Python，并且有一个建议的练习作业，基本上是做一个执行此操作的程序：
- 输入要在亚马逊搜索的产品
- 使用 requests.get() 和 .text() 获取该搜索页面的 html
- 使用 beautifulsoup 在 html 中搜索表示产品页面链接的 css 选择器
- 在单独的选项卡中，为您的搜索结果打开前五个产品的选项卡
这是我的代码：
#! python3
# Searches amazon for the inputted product (either through command line or input) and opens 5 tabs with the top
# items for that search.
import requests, sys, bs4, webbrowser
if len(sys.argv) > 1: # if there are system arguments
res = requests.get('https://www.amazon.com/s?k=' + ''.join(sys.argv))
res.raise_for_status
else: # take input
print('what product would you like to search Amazon for?')
product = str(input())
res = requests.get('https://www.amazon.com/s?k=' + ''.join(product))
res.raise_for_status
# retrieve top search links:
soup = bs4.BeautifulSoup(res.text, 'html.parser')
print(res.text) # TO CHECK HTML OF SITE, GET RID OF DURING ACTUAL PROGRAM
# open a new tab for the top 5 items, and get the css selector for links
# a list of all things on the downloaded page that are within the css selector 'a-link-normal a-text-normal'
linkElems = soup.select('a-link-normal a-text-normal')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
urlToOpen = 'https://www.amazon.com/' + linkElems[i].get('href')
print('Opening', urlToOpen)
webbrowser.open(urlToOpen)
我认为我选择了正确的 css 选择器（“a-link-normal a-text-normal”），所以我认为问题出在 res.text() - 当我打印以查看它的外观时例如，当我使用 chrome 中的检查元素查看同一站点时，html 内容似乎并不完整，或者包含实际 html 的内容。此外，这些 html 都不包含任何内容，例如“a-link-normal a-text-normal”。
仅作为示例，这是 res.text() 搜索“大铅笔”时的样子：
what product would you like to search Amazon for?
big pencil
<!--
To discuss automated access to Amazon data please contact api-services-support@amazon.com.
For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_5_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html/ref=rm_5_ac for advertising use cases.
-->
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title>Sorry! Something went wrong!</title>
<style>
html, body {
padding: 0;
margin: 0
}
img {
border: 0
}
#a {
background: #232f3e;
padding: 11px 11px 11px 192px
}
#b {
position: absolute;
left: 22px;
top: 12px
}
#c {
position: relative;
max-width: 800px;
padding: 0 40px 0 0
}
#e, #f {
height: 35px;
border: 0;
font-size: 1em
}
#e {
width: 100%;
margin: 0;
padding: 0 10px;
border-radius: 4px 0 0 4px
}
#f {
cursor: pointer;
background: #febd69;
font-weight: bold;
border-radius: 0 4px 4px 0;
-webkit-appearance: none;
position: absolute;
top: 0;
right: 0;
padding: 0 12px
}
@media (max-width: 500px) {
#a {
padding: 55px 10px 10px
}
#b {
left: 6px
}
}
#g {
text-align: center;
margin: 30px 0
}
#g img {
max-width: 90%
}
#d {
display: none
}
#d[src] {
display: inline
}
</style>
</head>
<body>
<a href="/ref=cs_503_logo"><img id="b" src="https://images-na.ssl-images-amazon.com/images/G/01/error/logo._TTD_.png" alt="Amazon.com"></a>
<form id="a" accept-charset="utf-8" action="/s" method="GET" role="search">
<div id="c">
<input id="e" name="field-keywords" placeholder="Search">
<input name="ref" type="hidden" value="cs_503_search">
<input id="f" type="submit" value="Go">
</div>
</form>
<div id="g">
<div><a href="/ref=cs_503_link"><img src="https://images-na.ssl-images-amazon.com/images/G/01/error/500_503.png"
alt="Sorry! Something went wrong on our end. Please go back and try again or go to Amazon's home page."></a>
</div>
<a href="/dogsofamazon/ref=cs_503_d" target="_blank" rel="noopener noreferrer"><img id="d" alt="Dogs of Amazon"></a>
<script>document.getElementById("d").src = "https://images-na.ssl-images-amazon.com/images/G/01/error/" + (Math.floor(Math.random() * 43) + 1) + "._TTD_.jpg";</script>
</div>
</body>
</html>
非常感谢您的耐心等待。
【问题讨论】：
标签： python web-scraping beautifulsoup