Debezium 服务器错误：java.lang.OutOfMemoryError：Java 堆空间答案

【问题标题】：Debezium server error: java.lang.OutOfMemoryError: Java heap spaceDebezium 服务器错误：java.lang.OutOfMemoryError：Java 堆空间
【发布时间】：2022-07-21 01:25:11
【问题描述】：

Debezium 服务器： v 1.9.0.Final

MongoDB Atlas： v 4.2.20

在带有 1GB CPU 和 4GB 内存的 Fargate 的 AWS ECS 上运行

概述： Debezium 启动初始快照并将一些数据发送到 kinesis，但在完成快照之前遇到错误（如下）。我尝试将容器的内存增加到 4GB，但不确定这是否是问题所在。我正在测试的一个集合是总共 28GB 和 1100 万个文档。

Debezium 配置（在 Terraform 中）：

      environment = [
        {
          "name" : "DEBEZIUM_SINK_TYPE",
          "value" : "kinesis"
        },
        {
          "name" : "DEBEZIUM_SINK_KINESIS_REGION",
          "value" : "us-east-1"
        },
        {
          "name" : "DEBEZIUM_SINK_KINESIS_CREDENTIALS_PROFILE",
          "value" : "default"
        },
        {
          "name" : "DEBEZIUM_SINK_KINESIS_ENDPOINT",
          "value" : "https://kinesis.us-east-1.amazonaws.com"
        },
        {
          "name" : "DEBEZIUM_SOURCE_CONNECTOR_CLASS",
          "value" : "io.debezium.connector.mongodb.MongoDbConnector"
        },
        {
          "name" : "DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME",
          "value" : "data/offsets.dat"
        },
        {
          "name" : "DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS",
          "value" : "0"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_NAME",
          "value" : "test"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_HOSTS",
          "value" : "test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_SSL_ENABLED",
          "value" : "true"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_MEMBERS_AUTO_DISCOVER",
          "value" : "true"
        },
        {
          "name" : "DEBEZIUM_SOURCE_DATABASE_INCLUDE_LIST",
          "value" : "test"
        },
        {
          "name" : "DEBEZIUM_SOURCE_COLLECTION_INCLUDE_LIST",
          "value" : "test.testCollection"
        },
        {
          "name" : "DEBEZIUM_SOURCE_CAPTURE_MODE",
          "value" : "change_streams_update_full"
        },
        {
          "name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY",
          "value" : "io.debezium.relational.history.FileDatabaseHistory"
        },
        {
          "name" : "DEBEZIUM_SOURCE_DATABASE_HISTORY_FILE_FILENAME",
          "value" : "history.dat"
        },
        {
          "name" : "QUARKUS_LOG_CONSOLE_JSON",
          "value" : "false"
        }
      ]
      secrets = [
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_USER",
          "valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:username::"
        },
        {
          "name" : "DEBEZIUM_SOURCE_MONGODB_PASSWORD",
          "valueFrom" : "${data.aws_secretsmanager_secret.test-debezium-read.arn}:password::"
        }
      ]

堆栈跟踪：

2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
    at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
    at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
    at java.base/java.lang.StringBuffer.append(StringBuffer.java:317)
    at java.base/java.io.StringWriter.write(StringWriter.java:106)
    at org.bson.json.StrictCharacterStreamJsonWriter.write(StrictCharacterStreamJsonWriter.java:368)
    at org.bson.json.StrictCharacterStreamJsonWriter.writeStartObject(StrictCharacterStreamJsonWriter.java:204)
    at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:22)
    at org.bson.json.LegacyExtendedJsonDateTimeConverter.convert(LegacyExtendedJsonDateTimeConverter.java:19)
    at org.bson.json.JsonWriter.doWriteDateTime(JsonWriter.java:129)
    at org.bson.AbstractBsonWriter.writeDateTime(AbstractBsonWriter.java:394)
    at org.bson.codecs.DateCodec.encode(DateCodec.java:32)
    at org.bson.codecs.DateCodec.encode(DateCodec.java:29)
    at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
    at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:203)
    at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
    at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
    at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
    at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:200)
    at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:217)
    at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:159)
    at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:46)
    at org.bson.Document.toJson(Document.java:453)
    at io.debezium.connector.mongodb.JsonSerialization.lambda$new$0(JsonSerialization.java:57)
    at io.debezium.connector.mongodb.JsonSerialization$$Lambda$521/0x0000000840448840.apply(Unknown Source)
    at io.debezium.connector.mongodb.JsonSerialization.getDocumentValue(JsonSerialization.java:89)
    at io.debezium.connector.mongodb.MongoDbSchema$$Lambda$580/0x00000008404ce840.apply(Unknown Source)
    at io.debezium.connector.mongodb.MongoDbCollectionSchema.valueFromDocumentOplog(MongoDbCollectionSchema.java:90)
    at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:68)
    at io.debezium.connector.mongodb.MongoDbChangeSnapshotOplogRecordEmitter.emitReadRecord(MongoDbChangeSnapshotOplogRecordEmitter.java:27)
    at io.debezium.pipeline.AbstractChangeRecordEmitter.emitChangeRecords(AbstractChangeRecordEmitter.java:42)
    at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:163)

我注意到在快照期间，发送的记录数和最后记录的偏移量似乎没有改变，而每条消息之间经过的时间变长了。这似乎是一个指数退避的事情，但我并不完全确定。

例子：

2022-06-01 16:20:37,789 INFO  [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Beginning snapshot of 'test-mongodb-shard-0' at {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:37,804 INFO  [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0)   Exporting data for collection 'test-mongodb-shard-0.test.testCollection'
2022-06-01 16:20:42,983 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 717 records sent during previous 00:00:06.159, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:20:57,417 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:14.434, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:05,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:16,624 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:19.207, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:21:35,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
2022-06-01 16:21:53,130 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:00:36.505, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:22:05,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:23:17,521 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:01:24.391, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:23:35,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:26:06,523 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:02:49.003, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:26:35,107 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:31:18,075 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:05:11.552, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:31:35,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 16:42:07,711 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:10:49.636, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 16:42:35,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:03:12,872 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:21:05.161, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:03:35,117 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 17:45:58,637 INFO  [io.deb.con.com.BaseSourceTask] (pool-7-thread-1) 2048 records sent during previous 00:42:45.765, last recorded offset: {sec=1654100437, ord=138, initsync=true, h=0}
2022-06-01 17:46:05,106 INFO  [io.deb.con.mon.ReplicaSetDiscovery] (debezium-mongodbconnector-test-replica-set-monitor) Checking current members of replica set at test-mongodb-shard-00-00.test.mongodb.net:27017,test-mongodb-shard-00-01.test.mongodb.net:27017,test-mongodb-shard-00-02.test.mongodb.net:27017,test-mongodb-i-00-00.test.mongodb.net:27017
...
2022-06-01 18:22:23,976 ERROR [io.deb.con.mon.MongoDbSnapshotChangeEventSource] (debezium-mongodbconnector-test-replicator-snapshot-0) Error while attempting to sync 'test-mongodb-shard-0.test.testCollection': : java.lang.OutOfMemoryError: Java heap space

【问题讨论】：

stackoverflow.com/questions/37335/…
你给任务分配多少内存？你有任何大的嵌套文档吗？请减少 max.batch.size 只是为了测试
@KafKafOwn 我给任务 4GB 内存，是的，肯定有一些大型嵌套文档。我现在正在使用 max_batch_size = 2048 和 max_queue_size = 4096 进行测试，但看起来结果是一样的。
2048 是我认为的默认值，你可以试试 100 之类的吗？
另一个问题，快照在您的用例中是必须的吗？

标签： mongodb amazon-kinesis debezium debezium-server

【解决方案1】：

除了将容器内存增加到 4GB 之外，您还可以设置bigger 堆大小，例如可以将最大和初始堆大小设置为 2GB：

  -Xms2048m -Xmx2048m

如果问题继续，请按照以下步骤操作：

使用参数 -XX:+HeapDumpOnOutOfMemoryError 启动 JVM，这将在程序进入 OOM 时为您提供堆转储。
使用visualVM之类的工具来分析获得的堆转储。这将有助于识别内存泄漏。

【讨论】：

【解决方案2】：

没有真正回答OP，只是想分享我的经验。我也偶尔会收到java.lang.OutOfMemoryError，想知道是什么原因造成的。

我的设置：

Debezium 1.9.5
Kafka 2.8
Docker container memory - 6Gi
Java heap - 4Gi both min and max
max.queue.size.in.bytes - 512Mi
max.batch.size - 16384

来自标准输出的错误：

2022-07-20T16:47:16.348943181Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.config"
2022-07-20T16:47:27.628395682Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.status"
2022-07-20T16:47:28.970536167Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | connector-producer-REDACTED-202207200823-0"
2022-07-20T16:47:33.787361085Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-3"
2022-07-20T16:47:45.067373810Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "server-timer"
2022-07-20T16:47:46.987669188Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | REDACTED-dbhistory"
2022-07-20T16:48:03.396881812Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-2"
2022-07-20T16:48:04.017710798Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-coordinator-heartbeat-thread | production"
2022-07-20T16:48:09.709036280Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "prometheus-http-1-3"
2022-07-20T16:48:14.667691706Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mysql-cj-abandoned-connection-cleanup"
2022-07-20T16:48:17.182623196Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-62"
2022-07-20T16:48:25.227925660Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-58"
2022-07-20T16:48:43.598026645Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP-Dispatcher"
2022-07-20T16:48:45.543984655Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-producer-network-thread | producer-1"
2022-07-20T16:48:52.284810255Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SourceTaskOffsetCommitter-1"
2022-07-20T16:48:56.992674380Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DistributedHerder-connect-1-1"
2022-07-20T16:49:18.691603140Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Session-HouseKeeper-47a3d56a-1"
2022-07-20T16:49:19.350459393Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "KafkaBasedLog Work Thread - cdc.offset"
2022-07-20T16:49:26.256350455Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | adminclient-8"
2022-07-20T16:49:33.154845201Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-59"
2022-07-20T16:49:34.414951745Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-60"
2022-07-20T16:49:40.871967276Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1890777616-61"
2022-07-20T16:49:56.007111292Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "debezium-sqlserverconnector-REDACTED-change-event-source-coordinator"
2022-07-20T16:50:00.410800756Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-admin-client-thread | connector-adminclient-REDACTED-0"

一些上下文：

我传输了一个每行约 1000 字节的表。有一个有用的max.queue.size.in.bytes 设置，我希望它可以作为连接器的上限，但我的仪表板显示队列大小最多只有 122Mi，大部分时间在 30Mi 和 60Mi 之间波动。我的计算是 1000*16384 应该永远不会达到 512Mi。

在这种特殊情况下，连接器已经正常工作了大约 3 个小时，然后有几分钟没有新的数据流。不久之后，OutOfMemory 出现了。

我还注意到，每次发生这种情况时，容器都会占用其允许的最大 CPU（在我的情况下为 4 个内核）。

【讨论】：