如何提取像这样的锚标签 <a class="a-no-hover-decoration" ... >？答案

【问题标题】：How to extract the anchor tag which is like this <a class="a-no-hover-decoration" ... >?如何提取像这样的锚标签 <a class="a-no-hover-decoration" ... >？
【发布时间】：2021-08-15 00:13:48
【问题描述】：

所以，

假设我在谷歌搜索“白俄罗斯”，一旦我们这样做，我们就会收到一些模型卡，如下图所示，

现在，如果您查看 HTML 的 Inspector，我们将看到这些卡片的 HREF 在锚标记内，就像下图一样，（...表示额外的东西）

<a class="a-no-hover-decoration" href="https://www.liquor.com/recipes/white-russian/" .....>

所以，我感兴趣的是从此类锚标记中提取该 href（如果它们存在用于搜索）。

我的尝试，

import requests
from bs4 import BeautifulSoup

req = requests.get("https://www.google.com/search?q=White+Russian")
soup = BeautifulSoup(req.text, 'html.parser')
soup.find_all("a", {"class": "a-no-hover-detection"}) # this returns Nothing

我对 web-scraping 有点陌生，所以非常感谢您的帮助。

我的第二个问题是，对于任何给定的随机搜索，如何检测我们有这样的模型卡与没有这样的卡？

谢谢。

【问题讨论】：

标签： python html css web-scraping beautifulsoup

【解决方案1】：

您还可以使用 SelectorGadgets Chrome 扩展程序直观地获取 CSS 选择器。

from bs4 import BeautifulSoup
import requests, lxml

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

response = requests.get('https://www.google.com/search?q=white russian', headers=headers).text
soup = BeautifulSoup(response, 'lxml')
# select() method: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#searching-by-css-class
for result in soup.select('.cv2VAd .a-no-hover-decoration'):
  link = result['href']
  print(link)

输出：

https://www.liquor.com/recipes/white-russian/
https://www.delish.com/cooking/recipe-ideas/a29091466/white-russian-cocktail-recipe/
https://www.kahlua.com/en-us/drinks/white-russian/

或者，您可以使用Google Search Engine Results API 来完成。这是一个付费 API，可免费试用 5,000 次搜索。

要集成的代码：

from serpapi import GoogleSearch

params = {
  "api_key": "YOUR_API_KEY",
  "engine": "google",
  "q": "White Russian",
  "google_domain": "google.com",
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results['recipes_results']:
  link = result['link']
  print(link)

输出：

https://www.liquor.com/recipes/white-russian/
https://www.delish.com/cooking/recipe-ideas/a29091466/white-russian-cocktail-recipe/
https://www.kahlua.com/en-us/drinks/white-russian/

免责声明，我为 SerpApi 工作。

【讨论】：

【解决方案2】：

要从 Google 的服务器获得正确的响应，请指定 User-Agent HTTP 标头和 hl=en 参数（以获取英文结果）。另外，类名是a-no-hover-decoration，而不是a-no-hover-detection：

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0"
}

params = {"q": "White Russian", "hl": "en"}

req = requests.get(
    "https://www.google.com/search", params=params, headers=headers
)
soup = BeautifulSoup(req.text, "html.parser")
for a in soup.find_all("a", {"class": "a-no-hover-decoration"}):
    print(a["href"])

打印：

https://www.liquor.com/recipes/white-russian/
https://www.delish.com/cooking/recipe-ideas/a29091466/white-russian-cocktail-recipe/
https://www.bbcgoodfood.com/recipes/white-russian

【讨论】：

非常感谢您的帮助！理解我的错误。