【发布时间】：2020-09-08 14:35:33
【问题描述】：
我正在尝试获取在此亚马逊页面前 3 页上出售的每台笔记本电脑的 URL
每次我运行脚本时，driver.findElements(By.xpath) 返回的 URL 数量不一致。第一页非常一致，它返回 4 个 URL，但第 2 页和第 3 页可以返回 1 到 4 个 URL 之间的任何位置，即使第 2 页有我要查找的 8 个 URL，第 3 页有我要查找的 4 个 URL。
我怀疑问题出在 GrabData 方法中，因为它根据给定的不一致 URL 列表抓取数据。我对此很陌生，所以我希望一切都有意义。任何帮助，将不胜感激。如果您需要更多说明，请告诉我
public static String dealURLsXpath = "//span[@data-a-strike=\"true\" or contains(@class,\"text-strike\")][.//text()]/parent::a[@class]";
public static List<String> URLs = new ArrayList<String>();
public static void main(String[] args)
{
//Initialize Browser
System.setProperty("webdriver.chrome.driver", "C:\\Users\\email\\eclipse-workspace\\ChromeDriver 81\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
//Search through laptops and starts at page 1
Search.searchLaptop(driver);
//Grabs data for each deal and updates Products List directly
listingsURL = driver.getCurrentUrl();
//updates the global URLs List with the URLs found by driver.findElements(By.xpath)
updateURLsList(driver);
//Iterates through each URL and grabs laptop information to add to products list
grabData(driver, URLs, "Laptop");
// Clears URLs list so that it can be populated by the URLs in the next page
URLs.clear();
// returns driver to Amazon page to click on "page 2" button to go to next page and repeat process
driver.get(listingsURL);
driver.findElement(By.xpath("//a [contains(@href,'pg_2')]")).click();
listingsURL = driver.getCurrentUrl();
updateURLsList(driver);
grabData(driver, URLs, "Laptop");
URLs.clear();
driver.get(listingsURL);
driver.findElement(By.xpath("//a [contains(@href,'pg_3')]")).click();
listingsURL = driver.getCurrentUrl();
updateURLsList(driver);
grabData(driver, URLs, "Laptop");
URLs.clear();
driver.get(listingsURL);
}
public static void updateURLsList(WebDriver driver)
{
//list of deals on amazon page
/////////////////////////////////////////////INCONSISTENT/////////////////////////////////////////////
List<WebElement> deals = driver.findElements(By.xpath(dealURLsXpath));
//////////////////////////////////////////////////////////////////////////////////////////////////////
System.out.println("Deals Size: " + deals.size());
for(WebElement element : deals)
{
URLs.add(element.getAttribute("href"));
}
System.out.println("URL List size: " + URLs.size());
deals.clear();
}
public static void grabData(WebDriver driver, List<String> URLs, String category)
{
for(String url : URLs)
{
driver.get(url);
String name = driver.findElement(By.xpath("//span [@id = \"productTitle\"]")).getText();
System.out.println("Name: " + name);
String price = driver.findElement(By.xpath("//span [@id = \"priceblock_ourprice\"]")).getText();
System.out.println("price: " + price);
String Xprice = driver.findElement(By.xpath("//span [@class = \"priceBlockStrikePriceString a-text-strike\"]")).getText();
System.out.println("Xprice: " + Xprice);
String picURL = driver.findElement(By.xpath("//img [@data-old-hires]")).getAttribute("src");
System.out.println("picURL: " + picURL);
BufferedImage img;
System.out.println("URL: " + url);
try
{
img = ImageIO.read(new URL(picURL));
products.add(new Product(
name,
Integer.parseInt(price.replaceAll("[^\\d.]", "").replace(".", "").replace(",", "")),
Integer.parseInt(Xprice.replaceAll("[^\\d.]", "").replace(".", "").replace(",", "")),
img,
category,
url));
}
catch(IOException e)
{
System.out.println("Error: " + e.getMessage());
}
}
【问题讨论】：
-
findElements 在至少找到 1 个项目时返回。您可以使用标准睡眠，或捕获 StaleElement 异常并重新运行。有关捕获陈旧元素的方法，请参阅此：github.com/pcalkins/browsermator/blob/master/src/browsermator/…
标签： java selenium selenium-webdriver xpath webdriverwait