禁用 CSS 时，HTMLUnit 不会执行 JavaScript 并加载所有内容答案

【问题标题】：HTMLUnit won't execute JavaScript and load all content while CSS is disabled禁用 CSS 时，HTMLUnit 不会执行 JavaScript 并加载所有内容
【发布时间】：2021-02-08 17:24:41
【问题描述】：

我一直在尝试使用 HTMLUnit 从 ethermine.org 上的矿工地址仪表板中抓取页面数据。一个示例 URL 是：

https://ethermine.org/miners/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard

调用webClient.getOptions().setCssEnabled(false); 会阻止javascript 加载表数据。此方法的文档是：

启用/禁用 CSS 支持。默认情况下，启用此属性。如果禁用 HtmlUnit 将不会下载链接的 css 文件，也不会触发相关的 onload/onerror 事件。

所以，我假设发生的事情是只有在成功加载 CSS 时才会执行 Javascript。对于我的用例来说，启用 CSS 会使页面加载时间过长。所以，我在问是否有办法让 Javascript 运行以从这个页面加载表数据，同时保持 CSS 被禁用。也许这可以通过触发 onload 事件来完成？如何才能做到这一点。谢谢

这里是代码和日志输出：

public class Main {


    public static void main(String[] args) {
        WebClient webClient = new WebClient();
        webClient.getOptions().setCssEnabled(false); // comment this line out and the active worker table data loads just fine
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());

        try {

            WebRequest request = new WebRequest(new URL("https://ethermine.org/miners/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard"));
            HtmlPage page = webClient.getPage(request);

            webClient.waitForBackgroundJavaScript(5000); // long enough for all javascript to execute

            System.out.println(page.asText());

        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

这是禁用 CSS 的输出：

Feb 08, 2021 5:17:09 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function get in object .
Feb 08, 2021 5:17:09 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function isEqualNode in object [object HTMLLinkElement].
Feb 08, 2021 5:17:10 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/chunk-vendors.391ed6f9.css
Feb 08, 2021 5:17:10 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/app.69e6b25a.css
Feb 08, 2021 5:17:10 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/Dashboard.eb03c99b.css
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-color" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-color" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-color" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-color" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Home - Ethermine - Ethereum (ETH) mining pool
ETH
ETH
ZEC
YEC
ETC
BEAM
RVN
unchecked
Home
Statistics
Luck
Support
Start Mining
News
News
Legal Notices
ImprintPrivacy PolicyTerms of Operation
Resources
Help Desk3rd Party AppsAPI
Pools
ethermine.orgetc.ethermine.orgzcash.flypool.orgycash.flypool.orgbeam.flypool.orgravencoin.flypool.org
Other Platforms
etherchain.orgpoolwatch.iozcha.inyec.zcha.inbeaconcha.in

这是启用 CSS 的日志输出：

Feb 08, 2021 5:18:02 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function get in object .
Feb 08, 2021 5:18:02 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function isEqualNode in object [object HTMLLinkElement].
Feb 08, 2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard
Feb 08, 2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard/payouts
Feb 08, 2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/poolStats
Feb 08, 2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/settings
Feb 08, 2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard/msg
Feb 08, 2021 5:18:07 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/chunk-vendors.391ed6f9.css
Feb 08, 2021 5:18:07 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/app.69e6b25a.css
Feb 08, 2021 5:18:07 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/Dashboard.eb03c99b.css
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-color" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-color" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-color" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-color" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08, 2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Home - Ethermine - Ethereum (ETH) mining pool
ETH
ZEC
YEC
ETC
BEAM
RVN
unchecked
Home
Statistics
Luck
Support
Start Mining
News
0x32885f190a878d6db68cbC2889a3FF3abE3C257C
Dashboard
Payouts
Settings
Help Desk
Auto-Updateunchecked
Workers Active / Inactive
0/0
Unpaid Balance
0.00012 ETH
Estimated Earnings
0.00016 ETH
Daily
Hashrate
Current (GH/s)
0.0
Average (GH/s)
0.0
Reported (GH/s)
0.0
Shares
Valid 99%
0
Stale 1%
0
Invalid
0
Created with Highcharts 7.1.2HashrateHashrateCurrent HashrateAverage HashrateReported Hashrate18:0020:0022:008. Feb02:0004:0006:0008:0010:0012:0014:0016:0010.0 GH/s10.2 GH/s10.4 GH/s10.6 GH/s10.8 GH/s
Created with Highcharts 7.1.2SharesWorkersShares & WorkersValid SharesWorkersStale Shares2021
Active Workers (57)
Name    Reported Hashrate   Current Hashrate    Valid Shares    Stale Shares    Invalid Shares  Last seen
1-1 183.7   188.8   168 3   0   8 minutes ago
1-2 183.8   188.8   168 3   0   9 minutes ago
1-3 183.8   161.8   145 1   0   8 minutes ago
1-4 183.8   179.9   160 3   0   8 minutes ago
10-1    192.2   182.2   162 3   0   9 minutes ago
10-2    194.5   192.9   173 1   0   8 minutes ago
10-3    192.3   196.3   176 1   0   8 minutes ago
10-4    189.3   209.2   185 5   0   9 minutes ago
11-1    192.0   179.2   158 5   0   9 minutes ago
11-2    168.9   149.9   133 3   0   8 minutes ago
11-3    194.8   177.7   158 3   0   8 minutes ago
11-4    198.6   163.7   146 2   0   9 minutes ago
12-1    190.9   207.8   187 0   0   9 minutes ago
12-2    163.9   169.9   151 3   0   8 minutes ago
12-3    170.1   188.5   169 1   0   9 minutes ago
12-4    170.2   145.9   130 2   0   8 minutes ago
13-1    167.9   182.2   162 3   0   8 minutes ago
13-2    182.4   157.4   141 1   0   8 minutes ago
13-3    186.4   187.0   167 2   0   8 minutes ago
13-4    166.2   156.3   140 1   0   9 minutes ago
14-1    202.2   185.8   164 5   0   8 minutes ago
14-2    269.4   283.3   253 3   0   8 minutes ago
15-1    265.2   245.7   214 11  0   8 minutes ago
15-2    167.8   186.7   168 0   0   8 minutes ago
15-3    161.3   154.4   139 0   0   8 minutes ago
2-1 181.5   164.4   146 3   0   8 minutes ago
2-2 174.7   181.4   162 2   0   8 minutes ago
2-3 188.3   188.4   167 4   0   8 minutes ago
2-4 188.3   198.1   175 5   0   8 minutes ago
3-1 192.6   170.3   152 2   0   8 minutes ago
3-2 183.7   179.6   161 1   0   9 minutes ago
3-3 188.3   189.6   170 1   0   8 minutes ago
3-4 164.4   162.5   143 5   0   9 minutes ago
4-1 168.4   150.0   135 0   0   8 minutes ago
4-2 192.7   191.1   172 0   0   8 minutes ago
4-3 192.5   211.1   190 0   0   8 minutes ago
4-4 192.4   182.9   164 1   0   8 minutes ago
5-1 168.4   176.6   157 3   0   9 minutes ago
5-2 192.5   210.3   188 2   0   9 minutes ago
5-3 171.3   183.2   161 6   0   8 minutes ago
5-4 193.5   178.5   160 1   0   8 minutes ago
6-1 140.4   145.2   130 1   0   8 minutes ago
6-2 190.9   181.1   163 0   0   8 minutes ago
6-3 191.6   206.7   186 0   0   8 minutes ago
6-4 190.4   185.1   164 4   0   8 minutes ago
7-1 194.5   186.3   167 1   0   8 minutes ago
7-2 193.2   179.2   160 2   0   8 minutes ago
7-3 197.2   208.1   186 2   0   8 minutes ago
7-4 194.7   187.0   167 2   0   8 minutes ago
8-1 193.1   188.5   169 1   0   8 minutes ago
8-2 192.3   187.0   167 2   0   8 minutes ago
8-3 146.0   145.4   127 6   0   8 minutes ago
8-4 196.4   218.1   195 2   0   8 minutes ago
9-1 194.6   207.8   187 0   0   8 minutes ago
9-2 191.7   167.8   151 0   0   10 minutes ago
9-3 152.4   151.4   135 2   0   9 minutes ago
9-4 194.2   207.7   185 3   0   9 minutes ago
Legal Notices
ImprintPrivacy PolicyTerms of Operation
Resources
Help Desk3rd Party AppsAPI
Pools
ethermine.orgetc.ethermine.orgzcash.flypool.orgycash.flypool.orgbeam.flypool.orgravencoin.flypool.org
Other Platforms
etherchain.orgpoolwatch.iozcha.inyec.zcha.inbeaconcha.in

【问题讨论】：

标签： java htmlunit

【解决方案1】：

所以，我假设发生的事情是 Javascript 仅在 CSS 成功加载时执行。

错了——没有这样的依赖。

也许 js 代码停止了，因为它依赖于一些 css 东西的存在。检查您的日志，请尝试添加

webClient.getOptions().setThrowExceptionOnScriptError(false);

在第一个错误时不停止 js 执行。如果这没有帮助，请在 GitHub 上打开一个问题。

【讨论】：