Web Scraping Python 无法在 button.click() 上加载 url

Question

CSV 文件包含所用国家/地区的名称。但是，在阿根廷之后，它无法恢复 url。它返回一个空字符串。

country,country_url
Afghanistan,https://openaq.org/#/locations?parameters=pm25&countries=AF&_k=tomib2
Algeria,https://openaq.org/#/locations?parameters=pm25&countries=DZ&_k=dcc8ra
Andorra,https://openaq.org/#/locations?parameters=pm25&countries=AD&_k=crspt2
Antigua and Barbuda,https://openaq.org/#/locations?parameters=pm25&countries=AG&_k=l5x5he
Argentina,https://openaq.org/#/locations?parameters=pm25&countries=AR&_k=962zxt
Australia,
Austria,
Bahrain,
Bangladesh,

country.csv 如下所示：

Afghanistan,Algeria,Andorra,Antigua and Barbuda,Argentina,Australia,Austria,Bahrain,Bangladesh,Belgium,Bermuda,Bosnia and Herzegovina,Brazil,

使用的代码是：

driver = webdriver.Chrome(options = options, executable_path = driver_path)
url = 'https://openaq.org/#/locations?parameters=pm25&_k=ggmrvm'
driver.get(url)
time.sleep(2)

# This function opens .csv file that we created at the first stage
# .csv file includes names of countries
with open('1Countries.csv', newline='') as f:
    reader = csv.reader(f)
    list_of_countries = list(reader)
    list_of_countries = list_of_countries[0]
    print(list_of_countries) # printing a list of countries

# Let's create Data Frame of the country & country_url
df = pd.DataFrame(columns=['country', 'country_url'])

# With this function we are generating urls for each country page
for country in list_of_countries[:92]:
    try:
        path = ('//span[contains(text(),' + '\"' + country + '\"' + ')]')
        # "path" is used to filter each country on the website by
        # iterating country names.
        next_button = driver.find_element_by_xpath(path)
        next_button.click()
        # Using "button.click" we are get on the page of next country
        time.sleep(2)
        country_url = (driver.current_url)
        # "country_url" is used to get the url of the current page
        next_button.click()
    except:
        country_url = None

    d = [{'country': country, 'country_url': country_url}]
    df = df.append(d)

我试过增加睡眠时间，不知道是什么原因导致的？

Answer 1

您面临的挑战是国家/地区列表是可滚动的：

您的代码在不显示时停止工作有点方便。

这是一个相对简单的解决方案 - 您需要将其滚动到视图中。我已经对您的代码进行了快速测试，以确认它可以正常工作。我删除了 CSV 部分，硬编码了一个位于列表下方的国家/地区，并且我拥有使其滚动查看的部分：

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
import time

def ScrollIntoView(element):
    actions = ActionChains(driver)
    actions.move_to_element(element).perform()


url = 'https://openaq.org/#/locations?parameters=pm25&_k=ggmrvm'
driver = webdriver.Chrome()
driver.get(url)
driver.implicitly_wait(10)

country = 'Bermuda'

path = ('//span[contains(text(),' + '\"' + country + '\"' + ')]')
next_button = driver.find_element_by_xpath(path)
ScrollIntoView(next_button)  # added this
next_button.click()
time.sleep(2)
country_url = (driver.current_url)
print(country_url)  # added this
next_button.click()

这是打印的输出：

https://openaq.org/#/locations?parameters=pm25&countries=BM&_k=7sp499

您乐意将其合并到您的解决方案中吗？ （只是说如果您需要更多支持）

如果您自己没有注意到的原因是try 掩盖了NotInteractableException。看看如何处理错误here

try 语句非常有用 - 但跟踪发生的时间也很好，以便您以后修复它们。从那个链接借用一些代码，你可以在你的 catch 中尝试这样的事情：

except:
    print("Unexpected error:", sys.exc_info()[0])