屏幕在鼠标悬停时使用 Scrapy 和 Selenium 抓取 Datepicker

Question

例如，我需要废弃 this 之类的页面，并且我正在使用 Scrapy + Seleninum 与日期选择器日历进行交互。

我意识到，如果某个日期可用，则工具提示上会显示价格，如果您将鼠标悬停在该日期上，则不会发生任何事情。

当您将鼠标悬停在可用日期时，我如何获取动态显示的价格以及我如何知道它是否可用？

Answer 1

由于页面的动态特性，解决问题的方法并不那么简单——您必须在这里和那里使用waits，而且捕捉点击或悬停时出现的动态组件的 HTML 是很棘手的。

这是导航到页面的完整工作代码，单击“签入”输入，等待日历加载并报告日历中每一天的可用性（它使用 @987654323 @class 来确定）。然后，它使用 move_to_element() 浏览器操作悬停每个单元格，等待工具提示并获取价格：

from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get("https://www.airbnb.pt/rooms/265820?check_in=2016-04-26&guests=1&check_out=2016-04-29")

# wait for the check in input to load
wait = WebDriverWait(driver, 10)
elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.book-it-panel input[name=checkin]")))
elem.click()

# wait for datepicker to load
wait.until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, '.ui-datepicker:not(.loading)'))
)

days = driver.find_elements_by_css_selector(".ui-datepicker table.ui-datepicker-calendar tr td")
for cell in days:
    day = cell.text.strip()
    if not day:
        continue

    if "ui-datepicker-unselectable" in cell.get_attribute("class"):
        status = "Unavailable"
    else:
        status = "Available"

    price = "n/a"
    if status == "Available":
        # hover the cell and wait for the tooltip
        ActionChains(driver).move_to_element(cell).perform()
        price = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.datepicker-tooltip'))).text

    print(day, status, price)

打印：

1 Unavailable n/a
2 Unavailable n/a
3 Unavailable n/a
4 Unavailable n/a
5 Unavailable n/a
6 Unavailable n/a
7 Unavailable n/a
8 Unavailable n/a
9 Unavailable n/a
10 Unavailable n/a
11 Unavailable n/a
12 Unavailable n/a
13 Available €40
14 Unavailable n/a
15 Unavailable n/a
16 Unavailable n/a
17 Unavailable n/a
18 Unavailable n/a
19 Available €36
20 Available €49
21 Unavailable n/a
22 Available €49
23 Unavailable n/a
24 Unavailable n/a
25 Available €40
26 Available €39
27 Available €35
28 Available €37
29 Available €37
30 Available €37

Answer 2

你好，求解答

关键提示：我们在日历上的点击事件后等待几秒钟，因为 java 脚本在日历打开后需要内部处理时间。

public static void main(String[] args) throws InterruptedException {


    System.setProperty("webdriver.chrome.driver","D:\\eclipseProject\\StackOverFlow\\chromedriver_win32 (1)\\chromedriver.exe");
    WebDriver driver = new ChromeDriver();
    driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
    driver.manage().window().maximize();
    Actions act = new Actions(driver);
    WebDriverWait wait = new WebDriverWait(driver,30);
    driver.get("https://www.airbnb.pt/rooms/265820?check_in=2016-04-26&guests=1&check_out=2016-04-29");

    // selecting firstdate picker -- check in
    driver.findElement(By.xpath("//*[@class='col-sm-6']/input")).click();
    // NOTE: we have to give sleep due to java-script takes internal processing time on calendar after it get opens 
    Thread.sleep(5000);
    // NOTE: calendar is not completely visible hence to make it visible
    // scroll a little bit down
    ((JavascriptExecutor) driver).executeScript("window.scrollBy(0,200)");

    // take all calendar dates inside the list
    List<WebElement> myhAvDates = driver.findElements(By.xpath("//*[@class='ui-datepicker-calendar']/tbody/tr/td/a[contains(@class, 'ui-state-default')]"));

    System.out.println("Size is "+myhAvDates.size());
    for(int i=0;i<myhAvDates.size();i++){
        wait.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//*[@class='ui-datepicker-calendar']/tbody/tr/td/a")));
        System.out.println("Available Date is  : " + myhAvDates.get(i).getText());
        act.moveToElement(myhAvDates.get(i)).build().perform();
        wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector(".datepicker-tooltip")));
        WebElement toolTipElement = driver.findElement(By.cssSelector(".datepicker-tooltip"));
        System.out.println("Available Date is  : " + myhAvDates.get(i).getText() + "==" +"And price is "+ toolTipElement.getText());
        myhAvDates = driver.findElements(By.xpath("//*[@class='ui-datepicker-calendar']/tbody/tr/td/a"));
    }
}

上面的代码会产生类似

的答案

Size is 10
Available Date is  : 13
Available Date is  : 13==And price is €40
Available Date is  : 19
Available Date is  : 19==And price is €36
Available Date is  : 20
Available Date is  : 20==And price is €49
Available Date is  : 22
Available Date is  : 22==And price is €49
Available Date is  : 25
Available Date is  : 25==And price is €40
Available Date is  : 26
Available Date is  : 26==And price is €39
Available Date is  : 27
Available Date is  : 27==And price is €35
Available Date is  : 28
Available Date is  : 28==And price is €37
Available Date is  : 29
Available Date is  : 29==And price is €37
Available Date is  : 30
Available Date is  : 30==And price is €37