OutOfMemoryError：请求的数组大小超过了 apache camel 中的 VM 限制

【问题标题】：OutOfMemoryError: Requested array size exceeds VM limit in apache camelOutOfMemoryError：请求的数组大小超过了 apache camel 中的 VM 限制
【发布时间】：2020-04-10 16:18:36
【问题描述】：

在我的应用程序中，正在使用 apache camel 处理一个 txt 文件，应用程序使用交换对象将整个文件读入字符串，并在每行上执行分隔符添加，并使用 apache camel 的交换对象将整个文件写入文件。我低于 OutOfMemoryError。 ***我观察到此异常发生在交换对象完成文件处理后以及应该将文件移动到另一个文件夹（端点）时* **。

我的应用程序支持 16GB 堆空间。处理前的文件大小为 685MB，添加分隔符后可能会增加 200MB。 有人能帮我一下吗？提前谢谢你。

from("file:<filepath>/processed/INPUT?filter=#fileFilter&delay=120000&exclusiveReadLockStrategy=#fileReadLockStrategy&delete=true&autoCreate=true")
                    .threads(10, 10)
                    .process(fileProcessor).log("Came out of file processor")
                    .recipientList(header(DESTIONATION_PATH));

它是一个从 INPUT 文件夹和 fileProcessor 读取文件的简单路径，通过在其中添加分隔符来处理文件内容。 DESTIONATION_PATH 是放置文件的路径。我可以看到它正在打印日志“来自文件处理器”，当文件即将发送到目标文件夹时出现以下异常。

Error has occurred: 
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
    at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1842) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:385) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:64) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
    Suppressed: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
        ... 26 common frames omitted
    ***Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit***
        at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
        at java.lang.StringCoding.encode(StringCoding.java:344)
        at java.lang.String.getBytes(String.java:918)
        at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198)
        at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55)
        at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63)
        at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299)
        at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188)
        at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115)
        at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333)
        at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305)
        at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
        at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80)
        at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
        at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638)
        at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
        at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173)
        at org.apache.camel.processor.RecipientList.process(RecipientList.java:133)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43)
        at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157)
        at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87)
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300) ~[na:1.8.0_171]
    at java.lang.StringCoding.encode(StringCoding.java:344) ~[na:1.8.0_171]
    at java.lang.String.getBytes(String.java:918) ~[na:1.8.0_171]
    at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]

【问题讨论】：

  • 您好！欢迎来到 StackOverflow。听起来你有一条使用太多内存的路线，导致整个事情崩溃。查看您的路线是什么样子（XML 或 Java DSL）会很有帮助。我们可能会基于此提出建议。很难阅读堆栈跟踪来确定您现有的代码做了什么以及如何改进它。
  • @AdamHawkes，感谢您的回复。路由基于 Java DSL。
  • 太好了，您确实需要将该 RouteBuilder Java DSL 包含在您的问题中。在你这样做之前，没有人可以帮助你。
  • @AdamHawkes 我已经添加了 Java DSL 路由代码。

标签： java apache-camel out-of-memory


【解决方案1】：

很抱歉在这里回复晚了。我能够解决上述问题。 当我仔细查看日志时，它在标准 java 类“StringCoding.java”中失败了。

我的代码中的罪魁祸首低于 2 行：

String content = exchange.getIn().getBody(String.class); 
//Add deleimters in each line content string & write back the whole string in file.
//derive new content string from original content
exchange.getOut().setBody(content);// Camel stores content in String & sends it via route using character array.

分析：

Java 在后台将字符串存储为字符数组。按照逻辑，我们在每行中添加了一些分隔符。 我观察到（原始文件中的字符数+添加的分隔符数）> Integer.MAX_VALUE。 所以，这里的java未能将整个字符串存储到字符数组中&骆驼不能把文件 在目标文件夹路径中。发生这种情况是因为修改文件中的字符数超出了最大整数大小。

解决方案：

File contentFile = exchange.getIn().getBody(File.class);
//Process the contentFile as file line by line & write the records back in file.
exchange.getOut().setBody(contentFile);// Camel stores content in File & sends it via route using InputStream.

我保留了上面的代码并且它有效，我们没有看到 OutofMemory 问题，因为骆驼在通过路由发送文件时没有使用数组。

感谢您的阅读。快乐编码...!!!

【讨论】：

    猜你喜欢
    • 1970-01-01
    • 2015-11-03
    • 2018-10-13
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode