谷歌抓取不返回任何描述或电子邮件

Question

我试图从每个 Google 搜索中获取描述和电子邮件，但它只返回标题和链接。我正在使用 Selenium 打开页面并使用 bs4 来抓取实际内容。

我究竟做错了什么？请帮忙。 谢谢！

soup = BeautifulSoup(driver.page_source,'lxml')
result_div = soup.find_all('div', attrs={'class': 'g'})


links = []
titles = []
descriptions = []
emails = []
phones = []

for r in result_div:
# Checks if each element is present, else, raise exception
    try:
    # link
        link = r.find('a', href=True)

    # title
        title = None
        title = r.find('h3')

        if isinstance(title,Tag):
            title = title.get_text()

    # desc
        description = None
        description = r.find('div', attrs={'class': 'IsZvec'})
        #description = r.find('span')
    

        if isinstance(description, Tag):
            description = description.get_text()
            print(description)
    # email

        email = r.find(text=re.compile(r'[A-Za-z0-9\.\+_-]+@[A-Za-z0-9\._-]+\.[a-zA-Z]*'))

Answer 1

要抓取 Google 搜索，您只能使用 Beautifulsoup webscraping 库而不使用 selenium webdriver，这将提高脚本的速度。

为了避免被 Google 屏蔽，如果使用 requests 可以是 rotate user-agent，例如，在 PC、移动和平板电脑之间以及浏览器之间切换，例如Chrome、Firefox、Safari、Edge 等，默认情况下请求库中的user-agent 是python-requests，因此网站可以理解它是发送请求的脚本。

要收集必要的信息（电子邮件、描述、标题、编号等），您可以使用 CSS 选择器搜索，使用 SelectorGadget Chrome 扩展程序在页面上很容易识别（如果网站是通过 JavaScript 呈现的，则并不总是能完美运行）。

import requests, re, json, lxml
from bs4 import BeautifulSoup

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
}

params = {
  'q': 'Facebook.com Dantist gmail.com',   # query
  'hl': 'en',                              # language
  'gl': 'us'                               # country of the search, US -> USA
}

html = requests.get(f'https://www.google.com/search',
                    headers=headers,
                    params=params).text
soup = BeautifulSoup(html, 'lxml')

data = []

for result in soup.select('.tF2Cxc'):
    title = result.select_one('.DKV0Md').text
    link = result.find('a')['href']
    snippet = result.select_one('.lyLwlc').text
       
    match_email = re.findall(r'[w.-]+@[w.-]+.w+', snippet)
    email = ''.join(match_email)

    # https://stackoverflow.com/a/3868861/15164646
    match_phone = re.findall(r'((?:+d{2}[-.s]??|d{4}[-.s]??)?(?:d{3}[-.s]??d{3}[-.s]??d{4}|(d{3})s*d{3}[-.s]??d{4}|d{3}[-.s]??d{4}))', snippet)
    phone = ''.join(match_phone)
    
    data.append({
        'Title': title,
        'Link': link,
        'Email': email if email else None,
        'Phone': phone if phone else None
    })

print(json.dumps(data, indent=2, ensure_ascii=False))

示例输出：

[
  {
    "Title": "Island Dental Associates | Franklin Square NY - Facebook",
    "Link": "https://www.facebook.com/IslandDentalAssociates/",
    "Email": "islanddentalassociatesny@gmail.com",
    "Phone": "(516) 271-0585"
  },
  {
    "Title": "Dental Bright | Houston TX - Facebook",
    "Link": "https://www.facebook.com/DentalBrightHouston/",
    "Email": "Dentalbrighttx@gmail.com",
    "Phone": "(713) 783-6060"
  },
  # ...
]

---

作为替代方案，您可以使用来自 SerpApi 的 Google Search Engine Results API。这是一个带有免费计划的付费 API。 不同之处在于它将绕过 Google 的块（包括 CAPTCHA），无需创建解析器和维护它。

代码示例：

from serpapi import GoogleSearch
import os, json, re

params = {
   "engine": "google",                         # search engine. Google, Bing, Yahoo, Naver, Baidu...
   "q": "Facebook.com Dantist gmail.com",      # search query
   "api_key": os.getenv('API_KEY')             # your serpapi api key
}
 
search = GoogleSearch(params)                  # where data extraction happens
results = search.get_dict()                    # JSON -> Python dictionary

data = []

for result in results['organic_results']:
   title = result['title']
   link = result['link']
   snippet = result['snippet']

   match_email = re.findall(r'[w.-]+@[w.-]+.w+', snippet)
   email = '
'.join(match_email)

   match_phone = re.findall(r'((?:+d{2}[-.s]??|d{4}[-.s]??)?(?:d{3}[-.s]??d{3}[-.s]??d{4}|(d{3})s*d{3}[-.s]??d{4}|d{3}[-.s]??d{4}))', snippet)
   phone = ''.join(match_phone)

   data.append({
     'title': title,
     'link': link,
     'email': email if email else None,
     'phone': phone if phone else None
   })

print(json.dumps(data, indent=2, ensure_ascii=False))

输出：

The answer is identical to the answer bs4.