Eclipse Ditto 在 EKS 上部署后抛出 akka 异常

【问题标题】：Eclipse Ditto throws akka exception after deployment on EKSEclipse Ditto 在 EKS 上部署后抛出 akka 异常
【发布时间】：2022-06-30 15:16:50
【问题描述】：

我在 EKS（舵图）上运行 Eclipse Ditto v2.5.0，几天后服务停止工作。它不会返回任何结果，也不会持久化新事物。我在日志中发现了以下内容：

2022-06-28T08:06:12+02:00 Caused by: akka.stream.RemoteStreamRefActorTerminatedException: [SourceRef-139] Remote partner [Actor[akka://ditto-cluster@10.20.87.204:2551/system/Materializers/StreamSupervisor-0/$$q2c-SinkRef-139#-1677314214]] has terminated unexpectedly and no clean completion/failure message was received (possible reasons: network partition or subscription timeout triggered termination of partner). Tearing down.
2022-06-28T08:06:12+02:00   at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1.onTimer(SourceRefImpl.scala:374)
2022-06-28T08:06:12+02:00   at akka.stream.stage.TimerGraphStageLogic.onInternalTimer(GraphStage.scala:1665)
2022-06-28T08:06:12+02:00   at akka.stream.stage.TimerGraphStageLogic.$anonfun$getTimerAsyncCallback$1(GraphStage.scala:1654)
2022-06-28T08:06:12+02:00   at akka.stream.stage.TimerGraphStageLogic.$anonfun$getTimerAsyncCallback$1$adapted(GraphStage.scala:1654)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:467)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:517)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:625)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:800)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:818)
2022-06-28T08:06:12+02:00   at akka.actor.Actor.aroundReceive(Actor.scala:537)
2022-06-28T08:06:12+02:00   at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:716)
2022-06-28T08:06:12+02:00   ... 10 common frames omitted
2022-06-28T08:06:12+02:00 2022-06-28 08:06:12,408 ERROR [] o.e.d.i.u.a.ThingsAggregatorProxyActor akka://ditto-cluster/user/gatewayRoot/proxy/aggregatorProxy - [retrieve-thing-response] Upstream failed.
2022-06-28T08:06:12+02:00 akka.stream.RemoteStreamRefActorTerminatedException: [SourceRef-137] Remote partner [Actor[akka://ditto-cluster@10.20.87.204:2551/system/Materializers/StreamSupervisor-0/$$m2c-SinkRef-137#934810721]] has terminated unexpectedly and no clean completion/failure message was received (possible reasons: network partition or subscription timeout triggered termination of partner). Tearing down.
2022-06-28T08:06:12+02:00   at akka.stream.impl.streamref.SourceRefStageImpl$$anon$1.onTimer(SourceRefImpl.scala:374)
2022-06-28T08:06:12+02:00   at akka.stream.stage.TimerGraphStageLogic.onInternalTimer(GraphStage.scala:1665)
2022-06-28T08:06:12+02:00   at akka.stream.stage.TimerGraphStageLogic.$anonfun$getTimerAsyncCallback$1(GraphStage.scala:1654)
2022-06-28T08:06:12+02:00   at akka.stream.stage.TimerGraphStageLogic.$anonfun$getTimerAsyncCallback$1$adapted(GraphStage.scala:1654)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:467)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:517)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:625)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:800)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:818)
2022-06-28T08:06:12+02:00   at akka.actor.Actor.aroundReceive(Actor.scala:537)
2022-06-28T08:06:12+02:00   at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:716)
2022-06-28T08:06:12+02:00   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2022-06-28T08:06:12+02:00   at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:800)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:818)
2022-06-28T08:06:12+02:00   at akka.actor.Actor.aroundReceive(Actor.scala:537)
2022-06-28T08:06:12+02:00   at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2022-06-28T08:06:12+02:00   at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:716)
2022-06-28T08:06:12+02:00   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2022-06-28T08:06:12+02:00   at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2022-06-28T08:06:12+02:00   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2022-06-28T08:06:12+02:00   at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2022-06-28T08:06:12+02:00   at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2022-06-28T08:06:12+02:00   at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2022-06-28T08:06:12+02:00   at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2022-06-28T08:06:12+02:00   at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2022-06-28T08:06:12+02:00   at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2022-06-28T08:06:12+02:00   at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2022-06-28T08:06:12+02:00 2022-06-28 08:06:12,410 ERROR [78dae9eb-4515-4513-9930-3060f7ba9652] o.e.d.g.s.e.a.HttpRequestActor akka://ditto-cluster/user/$Xe - Got <Status.Failure> when a command response was expected: <akka.stream.RemoteStreamRefActorTerminatedException: [SourceRef-137] Remote partner [Actor[akka://ditto-cluster@10.32.57.210:2551/system/Materializers/StreamSupervisor-0/$$m2c-SinkRef-137#934810721]] has terminated unexpectedly and no clean completion/failure message was received (possible reasons: network partition or subscription timeout triggered termination of partner). Tearing down.>!
2022-06-28T08:06:12+02:00 java.util.concurrent.CompletionException: akka.stream.RemoteStreamRefActorTerminatedException: [SourceRef-137] Remote partner [Actor[akka://ditto-cluster@10.32.57.210:2551/system/Materializers/StreamSupervisor-0/$$m2c-SinkRef-137#934810721]] has terminated unexpectedly and no clean completion/failure message was received (possible reasons: network partition or subscription timeout triggered termination of partner). Tearing down.
2022-06-28T08:06:12+02:00   at org.eclipse.ditto.gateway.service.endpoints.actors.AbstractHttpRequestActor.lambda$getResponseAwaitingBehavior$21(AbstractHttpRequestActor.java:387)
2022-06-28T08:06:12+02:00   at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2022-06-28T08:06:12+02:00   at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2022-06-28T08:06:12+02:00   at scala.PartialFunction.applyOrElse(PartialFunction.scala:214)

和

2022-06-27T16:22:19+02:00 2022-06-27 16:22:19,305 ERROR [] a.m.c.b.i.HttpContactPointBootstrap akka://ditto-cluster@12.25.88.222:2551/system/bootstrapCoordinator/contactPointProbe-10-20-68-87.ditto.pod.cluster.local-8558 - Overdue of probing-failure-timeout, stop probing, signaling that it's failed

我如何调试它并确定根本原因可能是什么？

【问题讨论】：

    标签： akka akka-http akka-cluster eclipse-ditto ditto


    【解决方案1】：

    登录表明您已尝试通过 HTTP 获取一些内容。 正如我们在中看到的，网关服务收到了这个错误：

    2022-06-28T08:06:12+02:00   at org.eclipse.ditto.gateway.service.endpoints.actors.AbstractHttpRequestActor.lambda$getResponseAwaitingBehavior$21(AbstractHttpRequestActor.java:387)

    ThingsAggregatorProxyActor 用于获取您从 EKS 中的 things 服务请求的每个 thing。

    1. 我会检查同上的健康端点。

      假设您在 EKS 中使用 nginx，您应该可以使用 localhost:30080/status/health下的devops用户>>>Source

      如果您不使用 nginx，只需调用网关 pod。 例如：网关：8080/status/health

    2. 检查 things pod 的日志，以及 pod 是否重新启动或有任何问题。

    【讨论】：

      猜你喜欢
      • 1970-01-01
      • 2012-03-25
      • 1970-01-01
      • 2017-01-11
      • 2021-11-30
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多
      热门标签
      Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode