尝试使用 beautifulsoup 从 google 中提取 url 时得到一个空列表答案

【问题标题】：getting an empty list when trying to extract urls from google with beautifulsoup尝试使用 beautifulsoup 从 google 中提取 url 时得到一个空列表
【发布时间】：2023-02-07 18:28:32
【问题描述】：

我正在尝试提取从谷歌位置搜索返回的前 100 个 url 但是我每次都得到一个空列表（“未找到结果”）

import requests
from bs4 import BeautifulSoup

def get_location_info(location):
    query = location + " information"
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36'
    }
    url = "https://www.google.com/search?q=" + query
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')
    results = soup.find_all("div", class_="r")
    websites = []
    if results:
        counter = 0
        for result in results:
            websites.append(result.find("a")["href"])
            counter += 1
            if counter == 100:
                break
    else:
        print("No search results found.")
    return websites

location = "Athens"
print(get_location_info(location))

没有找到搜索结果。 []

我也尝试过这种方法：

import requests
from bs4 import BeautifulSoup

def get_location_info(location):
    query = location + " information"
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36'
     }
    url = "https://www.google.com/search?q=" + query
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')
    results = soup.find_all("div", class_="r")
    websites = [result.find("a")["href"] for result in results][:10]
    return websites

location = "sifnos"
print(get_location_info(location))`

我得到一个空列表。我想我正在做类似帖子中建议的所有事情，但我仍然一无所获

【问题讨论】：

您能否展示您正在解析的 HTML 示例，其中包含 <div class="r"> 元素？我看了看谷歌结果，但没有看到，难道你正在搜索不存在的东西吗？

标签： python web-scraping beautifulsoup

【解决方案1】：

总是首先，看看你的汤，看看所有预期的成分是否都到位。

在这种情况下选择更具体的元素，例如 css selector：

[a.get('href') for a in soup.select('a:has(>h3)')]

要使同意横幅无效，还请发送一些cookies：

cookies={'CONSENT':'YES+'}

例子

import requests
from bs4 import BeautifulSoup

def get_location_info(location):
    query = location + " information"
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36'
     }
    url = "https://www.google.com/search?q=" + query
    response = requests.get(url, headers=headers, cookies={'CONSENT':'YES+'})
    soup = BeautifulSoup(response.text, 'html.parser')
    websites = [a.get('href') for a in soup.select('a:has(>h3)')]
    return websites

location = "sifnos"
print(get_location_info(location))

输出

['https://www.griechenland.de/sifnos/', 'http://de.sifnos-greece.com/plan-trip-to-sifnos/travel-information.php', 'https://www.sifnosisland.gr/', 'https://www.visitgreece.gr/islands/cyclades/sifnos/', 'http://www.griechenland-insel.de/Hauptseiten/sifnos.htm', 'https://worldonabudget.de/sifnos-griechenland/', 'https://goodmorningworld.de/sifnos-griechenland/', 'https://de.wikipedia.org/wiki/Sifnos', 'https://sifnos.gr/en/sifnos/', 'https://www.discovergreece.com/de/cyclades/sifnos']

【讨论】：