Java OutOfMemoryError / OrientDB WAL Flush Task 将 2M 记录导入 OrientDB 后出错

【问题标题】：Java OutOfMemoryError / OrientDB WAL Flush Task error after importing 2M records into OrientDBJava OutOfMemoryError / OrientDB WAL Flush Task 将 2M 记录导入 OrientDB 后出错
【发布时间】：2014-07-07 02:55:39
【问题描述】：

在我的一个导入脚本完成所有数据的导入后，我尝试重新启动它以获取任何更新的数据。它做的第一件事就是抓取最近更新的记录：

db.select().from(newClass).order('updatedAt desc').limit(1).one()

但是，这导致我的节点脚本出现以下错误：

Possibly unhandled OrientDB.RequestError: Java heap space
    at Operation.parseError (/Users/gsquare567/node_modules/oriento/lib/transport/binary/protocol/operation.js:779:13)
    at Operation.consume (/Users/gsquare567/node_modules/oriento/lib/transport/binary/protocol/operation.js:369:35)
    at Connection.process (/Users/gsquare567/node_modules/oriento/lib/transport/binary/connection.js:324:17)
    at Connection.handleSocketData (/Users/gsquare567/node_modules/oriento/lib/transport/binary/connection.js:250:17)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:748:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:410:10)
    at emitReadable (_stream_readable.js:406:5)
    at readableAddChunk (_stream_readable.js:168:9)

我收到以下服务器输出：

java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid1694.hprof ...
Heap dump file created [2055557443 bytes in 37.799 secs]
Error on fetching record during browsing. The record has been skipped
Error on retrieving record #11:1023466 (cluster: user)
-> com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:252)
-> com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:1017)
-> com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:65)
-> com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:264)
-> com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:40)
-> com.orientechnologies.orient.core.iterator.OIdentifiableIterator.readCurrentRecord(OIdentifiableIterator.java:285)
-> com.orientechnologies.orient.core.iterator.ORecordIteratorClusters.hasNext(ORecordIteratorClusters.java:139)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:913)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:397)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:358)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:60)
-> com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:94)
-> com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:83)
-> com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1181)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:340)
-> com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:169)
-> com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45)
GC overhead limit exceeded

我的另一个导入脚本同时运行，也因以下原因停止：

"error":{"name":"OrientDB.RequestError","message":"Java heap space","data":{},"previous":[],"id":1,"type":"java.lang.OutOfMemoryError","hasMore":0}

尝试再次运行原始脚本后，我在服务器中得到以下输出：

Exception in thread "OrientDB WAL Flush Task (pumpup)" Error on client connection
Java heap spacejava.lang.OutOfMemoryError: Java heap space

节点脚本输出：

Possibly unhandled OrientDB.RequestError: Java heap space
    at Operation.parseError (/Users/gsquare567/node_modules/oriento/lib/transport/binary/protocol/operation.js:779:13)
    at Operation.consume (/Users/gsquare567/node_modules/oriento/lib/transport/binary/protocol/operation.js:369:35)
    at Connection.process (/Users/gsquare567/node_modules/oriento/lib/transport/binary/connection.js:324:17)
    at Connection.handleSocketData (/Users/gsquare567/node_modules/oriento/lib/transport/binary/connection.js:250:17)
    at Socket.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:748:14)
    at Socket.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:410:10)
    at emitReadable (_stream_readable.js:406:5)
    at readableAddChunk (_stream_readable.js:168:9)

编辑

将内存限制增加到 2GB 后，我能够插入 5M 记录（而不是之前的 2M 记录），但仍然遇到此错误。

GC overhead limit exceeded
-> com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:252)
-> com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:1017)
-> com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:65)
-> com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:264)
-> com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:40)
-> com.orientechnologies.orient.core.iterator.OIdentifiableIterator.readCurrentRecord(OIdentifiableIterator.java:285)
-> com.orientechnologies.orient.core.iterator.ORecordIteratorClusters.hasNext(ORecordIteratorClusters.java:139)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:913)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:397)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:358)
-> com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:60)
-> com.orientechnologies.orient.core.storage.OStorageEmbedded.executeCommand(OStorageEmbedded.java:94)
-> com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:83)
-> com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:59)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1181)
-> com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:340)
-> com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:169)
-> com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:45)
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-0"

【问题讨论】：

  • 解决OOME一般有两种选择：增加可用内存或减少消耗
  • 你有多少内存？您可以在 bin/server.sh 文件中更改它，方法是将 -Xmx2G 添加到最后一行（Java 启动的地方），将 2G（最大）分配给 OrientDB。
  • 我 Luca，我已经做到了，这次我做到了 5M 行，但这仍然不够。我现在无法对我的数据运行任何查询，并且可能不得不重新创建我的数据库，因为这会导致 WAL 错误。怎么可能有数十亿条记录？

标签： java database out-of-memory orientdb wal


【解决方案1】：

这里的问题是您正在尝试对所有 5M 记录进行排序。

该操作必须将整个数据集加载到内存中才能对其进行排序（我们实际上有计划对其进行优化以避免在这种情况下出现 OOM，但尚未实现）。

因此，即使您指定限制 1，您也会加载整批记录，查询会很慢并且会消耗大量内存。

在 updatedAt 字段上优化该查询建立索引。

【讨论】：

  • 如果DATETIME 不是唯一的，您会推荐NOTUNIQUE_HASH_INDEX 吗？
  • 它适用于 NOTUNIQUE 而不是 NOTUNIQUE_HASH_INDEX。看了下文档，后者不支持范围查询，所以说得通。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2018-02-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode