不通过 Python3.8+Selenium+beautifulSoup 重定向到 Cookie 启用页面

Question

我正在使用以下代码使用 Python + Sellenium + BeautifulSoup4，但无法打开网站 (https://check.spamhaus.org/) 并对其进行刮擦。

我的代码是：

import requests
from bs4 import BeautifulSoup
url = "https://check.spamhaus.org/"

r = requests.get(url)
htmlContent = r.content
soup = BeautifulSoup(htmlContent, 'html.parser')
print(soup.prettify) 

请帮助通知 GAP。

Answer 1

该页面在允许您访问内容之前会进行一些机器人检测。因此，requests 包不会有任何帮助，因为r.content 会返回加载页面的内容。

您需要使用selenium 来实现您想要的。

url = "https://check.spamhaus.org/"

options = Options()
# Disable this switch to help prevent the website to detect an automated tool (selenium)
options.add_argument("--disable-blink-features=AutomationControlled")

with webdriver.Chrome(options=options) as driver:
    # This will open a new tab with the desired url
    driver.execute_script(f'''window.open("{url}","_blank");''')
    # Switch to the active window
    driver.switch_to.window(driver.window_handles[-1])

    # Waiting until the page is loaded (Validating the ipLookup button)
    wait = WebDriverWait(driver, 15)
    wait.until(EC.presence_of_element_located((By.ID, 'ipLookup')))

    soup = BeautifulSoup(driver.page_source, 'html.parser')
    print(soup.prettify)

您还需要添加以下导入：

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup

我没有使用driver.get(url)，而是使用window.open()，这实际上会打开一个带有所需网址的新标签。我无法 100% 解释原因，但使用 driver.get()，页面仍未通过加载页面。

至于您的评论问题，您可以在这里执行搜索

search_button = wait.until(EC.presence_of_element_located((By.ID, 'ipLookup')))

search_text_field = driver.find_element(By.CSS_SELECTOR, '[name="ip-search"]')
search_text_field.send_keys('1.1.1.1')
search_button.click()

Answer 2

我包含了来自@Nic 的answer 的一些信息，并使用了find_element_by_id(id) 和find_element_by_name(name)，事情开始对我有用。另外，在driver.switch_to.window 函数中，我将索引号更改为1。

这是我的代码：

#!/usr/bin/python3.8
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup

import time

url = "https://check.spamhaus.org/"

options = Options()
# Disable this switch to help prevent the website to detect an automated tool (selenium)
options.add_argument("--disable-blink-features=AutomationControlled")

with webdriver.Chrome(executable_path='/home/kkishore/Documents/doc/python-practise/updated-driver2/chromedriver',options=options) as driver:
    # This will open a new tab with the desired url
    driver.execute_script(f'''window.open("{url}","_blank");''')
    driver.switch_to.window(driver.window_handles[1])

    # Created by Kamal
    time.sleep(5)
    # INPUT the domain name 
    driver.find_element_by_name('ip-search').send_keys('example.com')
    # Click on lookup button
    driver.find_element_by_id('ipLookup').click()
    time.sleep(10)
##    soup = BeautifulSoup(driver.page_source, 'html.parser')
##    print(soup.prettify)