鼠标悬停后弹出时使用selenium和python提取数据

Question

大家好，这是我的第一个问题。我正在尝试从网站中提取数据。但问题是，它只有在我将鼠标悬停在它上面时才会出现。数据的网站是http://insideairbnb.com/melbourne/。当我将鼠标指针悬停在地图上的点上时，我想从弹出的面板中提取每个列表的入住率。我正在尝试使用此 stackoverflow 帖子 Scrape website with dynamic mouseover event 中的 @frianH 代码。我是使用硒进行数据提取的新手。我有关于 bs4 包的知识。我还没有成功找到正确的 xpath 来完成任务。先感谢您。到目前为止我的代码是

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
browser = webdriver.Chrome(options=chrome_options, executable_path='C:\\Users\\Kunal\\chromedriver.exe')
browser.get('http://insideairbnb.com/melbourne/')
browser.maximize_window()

#wait all circle
elements = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.XPATH, '//*[@id="map"]/div[1]/div[2]/div[2]/svg')))
table = browser.find_element_by_class_name('leaflet-zoom-animated')

#move perform -> to table
browser.execute_script("arguments[0].scrollIntoView(true);", table)

data = []
for circle in elements:
    #move perform -> to each circle
    ActionChains(browser).move_to_element(circle).perform()
    # wait change mouseover effect
    mouseover = WebDriverWait(browser, 30).until(EC.visibility_of_element_located((By.XPATH, '//*[@id="neighbourhoodBoundaries"]')))
    data.append(mouseover.text)

print(data[0])

谢谢你

Answer 1

所以我检查了一堆页面，它似乎对 selenium 自己的方法很抗拒，所以我们必须依赖 javascript。这是完整的代码-

from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
browser = webdriver.Chrome(options=chrome_options, executable_path='chromedriver.exe')
browser.get('http://insideairbnb.com/melbourne/')
browser.maximize_window()

# Set up a 30 seconds webdriver wait
explicit_wait30 = WebDriverWait(browser, 30)

try:
    # Wait for all circles to load
    circles = explicit_wait30.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'svg.leaflet-zoom-animated > g:nth-child(2) > circle')))
except TimeoutException:
    browser.refresh()

data = []
for circle in circles:
    # Execute mouseover on the element
    browser.execute_script("const mouseoverEvent = new Event('mouseover');arguments[0].dispatchEvent(mouseoverEvent)", circle)
    # Wait for the data to appear
    listing = explicit_wait30.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#listingHover')))
    # listing now contains the full element list - you can parse this yourself and add the necessary data to `data`
    .......
    # Close the listing
    browser.execute_script("arguments[0].click()", listing.find_element_by_tag_name('button'))

我也在使用 css 选择器而不是 XPATH。以下是流程的工作原理-

circles = explicit_wait30.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'svg.leaflet-zoom-animated > g:nth-child(2) > circle')))

这会等到所有圆圈都存在并将它们提取到circles。

请记住，页面加载圈子的速度非常慢，因此我设置了一个 try/except 块，以在 30 秒内未加载时自动刷新页面。随意更改此设置

现在我们必须遍历所有的圆圈-

for circle in circles:

接下来是在圆圈上模拟mouseover 事件，我们将使用 javascript 来完成此操作

这就是 javascript 的样子（注意 circle 指的是我们将从 selenium 传递的元素）

const mouseoverEvent = new Event('mouseover');
circle.dispatchEvent(mouseoverEvent)

这是通过selenium执行脚本的方式-

browser.execute_script("const mouseoverEvent = new Event('mouseover');arguments[0].dispatchEvent(mouseoverEvent)", circle)

现在我们必须等待列表出现-

listing = explicit_wait30.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '#listingHover')))

现在，您已经有了listing，它是一个还包含许多其他元素的元素，您现在可以非常轻松地提取每个元素，然后将它们存储在data 中。

如果您不关心以不同的方式提取每个元素，只需在listing 上执行.text 将导致类似这样的结果-

'Tanya\n(No other listings)\n23127829\nSerene room for a single person or a couple.\nGreater Dandenong\nPrivate room\n$37 income/month (est.)\n$46 /night\n4 night minimum\n10 nights/year (est.)\n2.7% occupancy rate (est.)\n0.1 reviews/month\n1 reviews\nlast: 20/02/2018\nLOW availability\n0 days/year (0%)\nclick listing on map to "pin" details'

就是这样，然后你可以将结果附加到data，你就完成了！