使用 HtmlUnit 的 java 内存不足错误答案

【问题标题】：Out of memory error in java with using HtmlUnit使用 HtmlUnit 的 java 内存不足错误
【发布时间】：2016-05-30 22:42:22
【问题描述】：

我使用 htmlUnit 和 java 构建了一个 web bot，它做了很多事情，但是在运行大约 10 分钟后，我总是收到以下错误。

程序处理一些文本文件，但下面的错误似乎与 HtmlUnit 解析 javascript 有关。

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at net.sourceforge.htmlunit.corejs.javascript.Parser.forLoop(Parser.java:1297)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:982)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statements(Parser.java:886)
at net.sourceforge.htmlunit.corejs.javascript.Parser.block(Parser.java:1699)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1031)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)
at net.sourceforge.htmlunit.corejs.javascript.Parser.ifStatement(Parser.java:1102)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:970)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)
at net.sourceforge.htmlunit.corejs.javascript.Parser.parseFunctionBody(Parser.java:662)
at net.sourceforge.htmlunit.corejs.javascript.Parser.function(Parser.java:827)
at net.sourceforge.htmlunit.corejs.javascript.Parser.primaryExpr(Parser.java:2746)
at net.sourceforge.htmlunit.corejs.javascript.Parser.memberExpr(Parser.java:2388)
at net.sourceforge.htmlunit.corejs.javascript.Parser.unaryExpr(Parser.java:2292)
at net.sourceforge.htmlunit.corejs.javascript.Parser.mulExpr(Parser.java:2219)
at net.sourceforge.htmlunit.corejs.javascript.Parser.addExpr(Parser.java:2205)
at net.sourceforge.htmlunit.corejs.javascript.Parser.shiftExpr(Parser.java:2188)
at net.sourceforge.htmlunit.corejs.javascript.Parser.relExpr(Parser.java:2165)
at net.sourceforge.htmlunit.corejs.javascript.Parser.eqExpr(Parser.java:2139)
at net.sourceforge.htmlunit.corejs.javascript.Parser.bitAndExpr(Parser.java:2130)
at net.sourceforge.htmlunit.corejs.javascript.Parser.bitXorExpr(Parser.java:2121)
at net.sourceforge.htmlunit.corejs.javascript.Parser.bitOrExpr(Parser.java:2112)
at net.sourceforge.htmlunit.corejs.javascript.Parser.andExpr(Parser.java:2103)
at net.sourceforge.htmlunit.corejs.javascript.Parser.orExpr(Parser.java:2094)
at net.sourceforge.htmlunit.corejs.javascript.Parser.condExpr(Parser.java:2060)
at net.sourceforge.htmlunit.corejs.javascript.Parser.assignExpr(Parser.java:2033)
at net.sourceforge.htmlunit.corejs.javascript.Parser.assignExpr(Parser.java:2044)
at net.sourceforge.htmlunit.corejs.javascript.Parser.expr(Parser.java:2023)
at net.sourceforge.htmlunit.corejs.javascript.Parser.nameOrLabel(Parser.java:1774)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1053)
at net.sourceforge.htmlunit.corejs.javascript.Parser.statement(Parser.java:926)

每当我使用 HtmlUnit 加载 javascript 页面时都会遇到大量错误，例如：

net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Expected argument of type object, but instead had type object (http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js#4)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3935)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3919)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3944)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError1(ScriptRuntime.java:3954)
at net.sourceforge.htmlunit.corejs.javascript.ScriptableObject.ensureScriptableObject(ScriptableObject.java:2193)
at net.sourceforge.htmlunit.corejs.javascript.NativeObject.execIdCall(NativeObject.java:303)
at net.sourceforge.htmlunit.corejs.javascript.IdFunctionObject.call(IdFunctionObject.java:93)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1536)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:5)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:3)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:3)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:4)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script.n(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at script(http://ca.classistatic.com/service-static/frontend-service/common.b28d6b2c3f10a7b5388a.js:1)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3286)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:827)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:939)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:836)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:812)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:997)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:399)
at com.gargoylesoftware.htmlunit.html.HtmlScript$3.execute(HtmlScript.java:277)
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:293)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:799)
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:756)
at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1039)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:252)
at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:198)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:271)
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:159)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:478)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:352)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:417)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:402)
at WebBotServicesAdID.replyToAd(WebBotServicesAdID.java:269)
at WebBotServicesAdID.main(WebBotServicesAdID.java:771)
======= EXCEPTION END ========

但是，这些错误似乎是不可避免的，它们不会导致程序停止。页面仍然被加载，我仍然可以做我需要做的事情，所以我总是忽略它们。

这个 OutOfMemoryError 是我忽略的那些错误累积的结果吗？

【问题讨论】：

标签： java htmlunit

【解决方案1】：

GC 错误：

GC 代表垃圾收集器，错误意味着 GC 正在努力工作并且几乎什么也没做。例如，here 和 here 已经讨论过此类错误。您可以尝试增加堆（“-Xmx1024m”、“-Xmx512m”等）或禁用错误验证（“-XX:-UseGCOverheadLimit”）。我认为您不应该使用其中任何一个选项。

如果您运行的是 linux，您可以尝试使用jmap 来检查堆中的实时数据，然后相应地优化您的代码。

JS 错误：

根据this 的回答，这些错误似乎是由脚本错误引起的。尝试给定的建议来禁用它们。

如果抛出这些错误并且没有正确处理，一些对象可能会留在堆中，从而导致 GC 问题。或者你可能有一些代码（例如循环）实例化了比它应该的更多的对象。

【讨论】：