【问题标题】:ActiveMQ Artemis Thread dumpActiveMQ Artemis 线程转储
【发布时间】:2020-11-02 16:04:24
【问题描述】:

我对在 docker 上运行的 ActiveMQ Artemis 版本 2.10.0 有疑问。这正是一个线程问题(下面的日志)。在日志中的某个位置,我看到了以下条目:

Component org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on path 0

之后有一个条目宣布代理进程将被杀死:

AMQ224079: The process for the virtual machine will be killed, as component org.apache.activemq.artemis.core.io.buffer.TimedBuffer@51b01960 is not responsive

后来,由于 id=2,线程 id 为 200(即设置为变量 thread-pool-max-size =200 scheduled-thread-pool-max-size= 20 的值,如 the docs 中所述),有一系列线程进程 id 的条目。

Artemis website上发现如果broker处于不稳定状态可以关闭。

为什么会这样?

日志:

 WARN  [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on path 0
 ERROR [org.apache.activemq.artemis.core.server] AMQ224079: The process for the virtual machine will be killed, as component org.apache.activemq.artemis.core.io.buffer.TimedBuffer@51b01960 is not responsive
 WARN  [org.apache.activemq.artemis.core.server] AMQ222199: Thread dump: *******************************************************************************
Complete Thread dump
"Reference Handler" Id=2 RUNNABLE
        at java.base@11.0.5/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
        at java.base@11.0.5/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
        at java.base@11.0.5/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)


"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@1d3b0b50
        at java.base@11.0.5/java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.ReferenceQueue$Lock@1d3b0b50
        at java.base@11.0.5/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
        at java.base@11.0.5/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
        at java.base@11.0.5/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)


"Signal Dispatcher" Id=4 RUNNABLE


"Common-Cleaner" Id=18 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@a9de548
        at java.base@11.0.5/java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.ReferenceQueue$Lock@a9de548
        at java.base@11.0.5/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
        at java.base@11.0.5/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
        at java.base@11.0.5/java.lang.Thread.run(Thread.java:834)
        at java.base@11.0.5/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:134)


"oneagentsubpathsender 1.189.238.20200508-120527" Id=20 RUNNABLE (in native)


"oneagentperiodiceventsmanaged" Id=21 RUNNABLE (in native)


"oneagentautosensor" Id=22 RUNNABLE


"oneagentperiodicurls" Id=23 RUNNABLE (in native)


"oneagentperiodicrequests" Id=24 RUNNABLE (in native)


"ActiveMQ Artemis Server Shutdown Timer" Id=27 TIMED_WAITING on java.util.TaskQueue@3cc6ec06
        at java.base@11.0.5/java.lang.Object.wait(Native Method)
        -  waiting on java.util.TaskQueue@3cc6ec06
        at java.base@11.0.5/java.util.TimerThread.mainLoop(Timer.java:553)
        at java.base@11.0.5/java.util.TimerThread.run(Timer.java:506)

"Thread-0 (-scheduled-threads)" Id=29 RUNNABLE
        at java.management@11.0.5/sun.management.ThreadImpl.dumpThreads0(Native Method)
        at java.management@11.0.5/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:502)
        at java.management@11.0.5/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:490)
        at org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47)
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:1022)
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$0(ActiveMQServerImpl.java:678)
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$127/0x00000001002df840.run(Unknown Source)
        at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireAction(CriticalAnalyzerImpl.java:155)
        at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:140)
        at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53)
        at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284)
        at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$3.run(ActiveMQScheduledComponent.java:294)
        at java.base@11.0.5/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base@11.0.5/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base@11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

        Number of locked synchronizers = 1
        - java.util.concurrent.ThreadPoolExecutor$Worker@1b58ff9e


"Thread-0 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@466cf502)" Id=35 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.5/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)



"Thread-0 (activemq-netty-threads)" Id=112 WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50ab59d6
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50ab59d6
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
        at java.base@11.0.5/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:959)
        at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:835)
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.addToPage(AbstractJournalStorageManager.java:2074)
        at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1308)
        at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:1003)
        at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:894)
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2073)
        -  locked org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@7805129
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1712)
        -  locked org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@7805129
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1651)
        -  locked org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@7805129
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1643)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.lambda$sendShouldBlockProducer$0(AMQSession.java:453)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession$$Lambda$274/0x0000000100597840.run(Unknown Source)
        at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.checkMemory(PagingStoreImpl.java:728)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.sendShouldBlockProducer(AMQSession.java:504)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.send(AMQSession.java:415)
        at org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection$CommandProcessor.processMessage(OpenWireConnection.java:1570)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
        at org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection.bufferReceived(OpenWireConnection.java:293)
        at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
        at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-29 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@466cf502)" Id=200 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.5/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Scheduler-143999341" Id=202 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a25f629
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a25f629
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
        at java.base@11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.5/java.lang.Thread.run(Thread.java:834)


"Thread-1 (activemq-netty-threads)" Id=2344 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

【问题讨论】:

  • 为什么会这样?

标签: activemq-artemis


【解决方案1】:

据我所知,这个问题实际上与线程无关。正如错误消息所示,问题在于“TimedBuffer 在路径 0 上已过期”,或者换句话说,“TimedBuffer@51b01960 没有响应”。 “TimedBuffer”是负责将数据刷新到磁盘的组件,刷新数据需要很长时间,因此代理的“关键分析器”会关闭代理并发出线程转储以进行调试。

“关键分析器”是代理中的一项服务,用于监控重要任务,如果这些任务完成时间过长,那么关键分析器将采取行动。默认broker.xml 包含关键分析器的此配置:

<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>

因此,关键分析器每 60 秒将检查各种组件,如果其中任何一个已停止 120 秒(即 2 分钟),那么它将停止代理。您将需要手动重新启动代理,或者根据您的平台,如果代理充当“服务”,那么它可能会自动重新启动。您可以在the documentation 中阅读有关关键分析器的更多信息。

【讨论】:

  • HALT 是否意味着代理将被停止或重新启动?此外,TimedBuffer 将数据刷新到磁盘。这部分 HA 使用共享文件很痛吗?我最近出现了这个错误,正在尝试了解原因。
  • 我更新了我的答案以解决您评论的第一部分。我不太确定你在问什么关于 HA 的问题。 TimedBuffer 用于将所有数据刷新到磁盘,无论是在本地还是在远程共享磁盘上。希望有帮助!
  • 谢谢。我有这些默认设置,并且在 systemctl 上启用了 Artemis。昨晚出现了这个错误,看来Artemis自己重启了。对于 TimedBuffer,我试图了解此错误是否是由于我们用于数据文件夹的 NFS 挂载文件系统,用于 HA。
  • 需要明确的是,平台必须重新启动代理,因为在这种情况下,Artemis 本身没有任何重新启动功能。我仍然不确定我是否解决了您关于 HA/NFS 的问题。
猜你喜欢
  • 1970-01-01
  • 2018-06-30
  • 2020-10-09
  • 2021-02-15
  • 1970-01-01
  • 2021-12-21
  • 2021-10-09
  • 2022-01-04
  • 2021-10-13
相关资源
最近更新 更多