【发布时间】：2019-09-16 14:56:30
【问题描述】：
我使用 Kafka Streams 进行数据处理。
卡夫卡 1.1
我在 kafka 中生成 35k rps 的数据速率，并且流的消耗速度也与通过 kafka 消费者指标看到的相同。但我经常看到 reset partition errors 表示提取偏移量超出范围。这基本上意味着我的消费者比正在发生的非常积极的日志删除要慢。
我的日志保留小时数为 168，日志保留字节数为 10G。以下是我经常在消费者中获取的日志。
我试图在网上搜索这个相关问题，但我没有找到。
[sample-app-deploy-8c4fd5697-4xxbk sample-app] 09:28:24.291 [sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1-consumer, groupId=sample-app] Fetch offset 116411050 is out of range for partition sample-topic-4, resetting offset
[sample-app-deploy-8c4fd5697-4xxbk sample-app] 09:28:24.292 [sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-4 to offset 116411058.
[sample-app-deploy-8c4fd5697-qmjnd sample-app] 09:28:24.306 [sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1-consumer, groupId=sample-app] Fetch offset 237000869 is out of range for partition sample-topic-7, resetting offset
[sample-app-deploy-8c4fd5697-qmjnd sample-app] 09:28:24.307 [sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-7 to offset 237000871.
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:29:56.808 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Fetch offset 471945398 is out of range for partition sample-topic-0, resetting offset
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:29:56.810 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-0 to offset 471945403.
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:34:56.804 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Fetch offset 474036996 is out of range for partition sample-topic-0, resetting offset
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:34:56.805 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-0 to offset 474036997.
[sample-app-deploy-8c4fd5697-cjccm sample-app] 09:39:10.659 [sample-app-a7e7c388-0dd4-45e8-8d5a-3a84effb7dfd-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-a7e7c388-0dd4-45e8-8d5a-3a84effb7dfd-StreamThread-1-consumer, groupId=sample-app] Fetch offset 236702790 is out of range for partition sample-topic-5, resetting offset
谁能帮忙指出问题所在的区域？
【问题讨论】：
-
你们的清理政策是什么？
-
@SantoshTulasiram 看起来有类似的问题issues.apache.org/jira/browse/KAFKA-6189 你能检查一下它是否相关吗？
-
清理策略为删除。我发现由于保留时间而不是我预期的保留大小，清理工作正在积极进行。似乎我们正在推动的活动时间有问题。问题解决后我会在这里更新。
-
问题出在时间戳上。我们以秒而不是毫秒为单位发送事件时间，这导致时间保留非常激进。
-
@SantoshTulasiram 你能回答你的问题并接受吗？谢谢。
标签： apache-kafka apache-kafka-streams