该网站未在 chromedriver/geckodriver (Selenium) 中显示信息

Question

我一直在尝试废弃https://wizzair.com/en-gb/flights/timetable#/。 有一段时间进展顺利。但是从今天开始，我无法获取航班信息，因为Wizz服务器没有返回任何内容，并出现“发生错误。请重试。如果错误仍然存​​在，请联系航空公司。”错误。

我试图不是从 Selenium 而是从 .exe 手动访问该网站，同样的事情，对于 geckodriver 和 chrome 的。 所以，网站似乎知道它是一个自动管理的工具，并没有返回任何信息。

你有什么建议可以解决吗？

谢谢，

更新： 可以在此处找到已保存的网页，无论是否使用 WebDriver 都可以访问： https://drive.google.com/drive/folders/1OsqfKqKyqpOLBMdUbunYH7GUQZgqtRXJ?usp=sharing

代码试用

    System.setProperty("webdriver.gecko.driver", ".\\resources\\drivers\\geckodriver.exe");
    System.setProperty(FirefoxDriver.SystemProperty.DRIVER_USE_MARIONETTE, "true");
    System.setProperty(FirefoxDriver.SystemProperty.BROWSER_LOGFILE, "/dev/null");

    WebDriver driver = new FirefoxDriver();
    driver.manage().window().setPosition(new Point(2000, 0)); // move window to the second display
    driver.manage().window().maximize();
    driver.get("https://wizzair.com/en-gb/flights/timetable/clujnapoca/vienna--#/1/0/1/0/0/2019-01/2019-01");

Answer 1

您可以尝试更改 HTTP 请求中的标头，或者更改某些系统属性以隐藏您正在使用 selenium 浏览器的事实。

设置系统属性：

System.setProperty(property, value);

要查看您的 JVM 支持哪些属性，请打开控制台（CMD/终端）并输入：

java -XshowSettings:all

设置系统属性示例：

static {
    System.setProperty("user.dir", "C:\\Users\\YourName");
}

系统属性参考： https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html

---

屏蔽您与网站的连接：

注意： URLConnection#addRequestProperty 是您设置要发送到网站的标头的方式。您可以使用HTTP 或HTTPS，它们与URLConnection 的通用包相同或非常接近。

private static final String LINUX_USER_AGENT =
        "Mozilla/5.0 (X11; Linux x86_64; rv:52) Gecko/20100101 Firefox/62" + ".0";
private static final String WINDOWS_USER_AGENT =
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) " + "Gecko/20100101 Firefox/62.0";

/**
 * Masks the URL connection as a regular one would be 403 forbidden
 *
 * @param url
 *      - URL to mask and connect to
 * @return the masked url connection to the website
 *
 * @throws IOException
 */
public static InputStreamReader getMaskedInputStream(String url) throws IOException
{
    URL website = new URL(url);
    URLConnection connection = website.openConnection();
    if (System.getProperty("os.name").contains("Win"))
    {
        connection.addRequestProperty("User-Agent", WINDOWS_USER_AGENT);
    }
    else
    {
        connection.addRequestProperty("User-Agent", LINUX_USER_AGENT);
    }
    connection.addRequestProperty("Accept-Language", "en-US,en;q=0.5");
    connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
    return new InputStreamReader(connection.getInputStream());
}

/**
 * Masks the URL connection as a regular one would be 403 forbidden
 *
 * @param url
 *      - URL to mask and connect to
 * @return the masked url connection to the website
 *
 * @throws IOException
 */
public static InputStreamReader getMaskedInputStream(Proxy proxy, Authenticator auth, String url) throws IOException
{
    Authenticator.setDefault(auth);
    final URL website = new URL(url);
    final URLConnection connection = website.openConnection(proxy);
    if (System.getProperty("os.name").contains("Win"))
    {
        connection.addRequestProperty("User-Agent", WINDOWS_USER_AGENT);
    }
    else
    {
        connection.addRequestProperty("User-Agent", LINUX_USER_AGENT);
    }
    connection.addRequestProperty("Accept-Language", "en-US,en;q=0.5");
    connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
    return new InputStreamReader(connection.getInputStream());
}