【问题标题】:Amazon MQ High Storage PercentageAmazon MQ 高存储百分比
【发布时间】:2022-11-17 04:21:48
【问题描述】:

我们目前在我们的架构中使用活动/备用 Amazon MQ (ActiveMQ) 代理。我们注意到,随着时间的推移,我们的存储一直在不断增加,这会降低我们的代理性能并导致重启时间过长。我们目前使用 VirtualTopics,每个主题有 4 个消费者

我检查了队列,没有一个队列有大量待处理的消息。消费者都足够快地获取消息并处理它们。唯一拥有大约 40,000 条消息的队列是 DLQ。但是,我们也尝试删除该队列中的消息,使用的存储空间仍然没有减少。

我已联系亚马逊支持,但没有收到有用的答复。

我只想了解是否有一种方法可以了解哪些队列/主题正在占用存储空间,以便我们可以清除消息并减少存储空间以获得更好的性能和更快的重启。

我们的配置

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<broker schedulePeriodForDestinationPurge="10000" xmlns="http://activemq.apache.org/schema/core">
  <!--
  A configuration contains all of the settings for your ActiveMQ broker, in XML format (similar to ActiveMQ's activemq.xml file).
  You can create a configuration before creating any brokers. You can then apply the configuration to one or more brokers.

  You can use additional attributes for the broker element above. These attributes allow you to configure broker-wide settings.

  For more information, see Configuration and Amazon MQ Broker Configuration Parameters in the Amazon MQ Developer Guide:
  https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/amazon-mq-broker-configuration-parameters.html
  -->
  <!--
  Mirrored queues let you send a copy of each message to a topic with a similar name automatically.
  For more information, see http://activemq.apache.org/mirrored-queues.html

  Virtual destinations let you configure advanced routing of messages between destinations.
  For more information, see http://activemq.apache.org/virtual-destinations.html
  -->
  <!--
  <destinationInterceptors>
    <mirroredQueue copyMessage="true" postfix=".qmirror" prefix=""/>
    <virtualDestinationInterceptor>
      <virtualDestinations>
        <virtualTopic name="&gt;" prefix="VirtualTopicConsumers.*." selectorAware="false"/>
        <compositeQueue name="MY.QUEUE">
          <forwardTo>
            <queue physicalName="FOO"/>
            <topic physicalName="BAR"/>
          </forwardTo>
        </compositeQueue>
      </virtualDestinations>
    </virtualDestinationInterceptor>
  </destinationInterceptors>
  -->
  <!--
  By default, Amazon MQ optimizes for queues with fast consumers:
  Consumers are considered fast if they are able to keep up with the rate of messages generated by producers.
  Consumers are considered slow if a queue builds up a backlog of unacknowledged messages, potentially causing a decrease in producer throughput.
  To instruct Amazon MQ to optimize for queues with slow consumers, set the concurrentStoreAndDispatchQueues attribute to false.
  For more information, see https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/ensuring-effective-amazon-mq-performance.html
  -->
  <persistenceAdapter>
    <kahaDB concurrentStoreAndDispatchQueues="false"/>
  </persistenceAdapter>
  <destinationPolicy>
    <policyMap>
      <policyEntries>
        <!--
        gcInactiveDestinations is used to automatically purge inactive destinations
        preventing them from unnecessarily using broker resources.

        An 'inactive' destination is one that has no messages pending and no consumers connected.

        For more information, see: http://activemq.apache.org/delete-inactive-destinations.html
        -->
        <policyEntry gcInactiveDestinations="true" inactiveTimoutBeforeGC="600000" topic="&gt;">
          <!--
          The constantPendingMessageLimitStrategy is used to prevent
          slow topic consumers to block producers and affect other consumers
          by limiting the number of messages that are retained

          For more information, see: http://activemq.apache.org/slow-consumer-handling.html
          -->
          <pendingMessageLimitStrategy>
            <constantPendingMessageLimitStrategy limit="1000"/>
          </pendingMessageLimitStrategy>
        </policyEntry>
        <policyEntry gcInactiveDestinations="true" inactiveTimoutBeforeGC="600000" queue="&gt;"/>
        <!--
        Destination policies let you configure a rich set of behaviors for your queues and topics.
        For more information, see http://activemq.apache.org/per-destination-policies.html
        -->
        <!--
        <policyEntry topic="FOO.&gt;">
          <dispatchPolicy>
            <roundRobinDispatchPolicy/>
          </dispatchPolicy>
          <subscriptionRecoveryPolicy>
            <lastImageSubscriptionRecoveryPolicy/>
          </subscriptionRecoveryPolicy>
        </policyEntry>
        <policyEntry advisoryForConsumed="true" tempTopic="true"/>
        <policyEntry advisoryForConsumed="true" tempQueue="true"/>
        -->
      </policyEntries>
    </policyMap>
  </destinationPolicy>
  <!--
  Typically, destinations are created automatically when they are used. Amazon MQ lets you create destinations when the broker is started.
  For more information, see http://activemq.apache.org/configure-startup-destinations.html
  -->
  <!--
  <destinations>
    <queue physicalName="FOO.BAR"/>
    <topic physicalName="SOME.TOPIC"/>
  </destinations>
  -->
  <!--
  You can control advanced ActiveMQ features using plugins.
  -->
  <plugins>
    <!--
    The Authorization plugin allows you to control the groups of users that are allowed to perform certain operations on your destinations.
    For more information, see http://activemq.apache.org/security.html
    -->
    <!--
    <authorizationPlugin>
      <map>
        <authorizationMap>
          <authorizationEntries>
            <authorizationEntry admin="guests,users" queue="GUEST.&gt;" read="guests" write="guests,users"/>
            <authorizationEntry admin="guests,users" read="guests,users" topic="ActiveMQ.Advisory.&gt;" write="guests,users"/>
          </authorizationEntries>
          <tempDestinationAuthorizationEntry>
            <tempDestinationAuthorizationEntry admin="tempDestinationAdmins" read="tempDestinationAdmins" write="tempDestinationAdmins"/>
          </tempDestinationAuthorizationEntry>
        </authorizationMap>
      </map>
    </authorizationPlugin>
    -->
    <!--
    The Discarding DLQ plugin simplifies the configuration of your global dead-letter queue strategy.
    You can take advantage of a more granular per-destination control by using destination policies.
    For more information, see http://activemq.apache.org/message-redelivery-and-dlq-handling.html
    -->
    <!--
    <discardingDLQBrokerPlugin dropAll="true" dropTemporaryQueues="true" dropTemporaryTopics="true"/>
    -->
    <!--
    The Force Persistency Mode plugin can override the persistency mode set on messages.
    -->
    <!--
    <forcePersistencyModeBrokerPlugin persistenceFlag="true"/>
    -->
    <!--
    The Redelivery plugin extends the capabilities of destination policies with respect to message redelivery.
    For more information, see http://activemq.apache.org/message-redelivery-and-dlq-handling.html
    -->
    <!--
    <redeliveryPlugin fallbackToDeadLetter="true" sendToDlqIfMaxRetriesExceeded="true">
      <redeliveryPolicyMap>
        <redeliveryPolicyMap>
          <redeliveryPolicyEntries>
            <redeliveryPolicy maximumRedeliveries="4" queue="SpecialQueue" redeliveryDelay="60000"/>
          </redeliveryPolicyEntries>
          <defaultEntry>
            <redeliveryPolicy initialRedeliveryDelay="45000" maximumRedeliveries="4" redeliveryDelay="60000"/>
          </defaultEntry>
        </redeliveryPolicyMap>
      </redeliveryPolicyMap>
    </redeliveryPlugin>
    -->
    <!--
    The Statistics plugin lets you query broker or destination statistics by sending messages to the broker.
    For more information, see http://activemq.apache.org/statisticsplugin.html
    -->
    <!--
    <statisticsBrokerPlugin/>
    -->
    <!--
    The Timestamping plugin lets the broker use server-side time instead of client-provided time for messages.
    For more information, see http://activemq.apache.org/timestampplugin.html
    -->
    <!--
    <timeStampingBrokerPlugin ttlCeiling="86400000" zeroExpirationOverride="86400000"/>
    -->
  </plugins>
  <!--
  Network connectors let you connect brokers into networks of brokers.
  For more information, see Creating and Configuring an Amazon MQ Network of Brokers
  (https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/amazon-mq-creating-configuring-network-of-brokers.html)
  in the Amazon MQ Developer Guide and also Networks of Brokers
  (http://activemq.apache.org/networks-of-brokers.html) in the ActiveMQ documentation.
  -->
  <!--
  <networkConnectors>
    <networkConnector name="myNetworkConnector" userName="commonUser" uri="masterslave:(ssl://b-1a2b3c4d-1.mq.region.amazonaws.com:61617,ssl://b-1a2b3c4d-2.mq.region.amazonaws.com:61617)"/>
  </networkConnectors>
  -->
</broker>

【问题讨论】:

  • 亚马逊提供了什么没有用的答案?

标签: activemq amazon-mq


【解决方案1】:

很有可能是 DLQ 中的 40k 条消息。不要将您的消息代理用作长期存储;这不是它的意思。

由于一个过于简化的 KahaDB 写入了一堆数据条带,然后当消息从每个数据条带中被消费时,它可以 GC 一个数据条带。一旦存储在该条带中的所有消息都被消耗,它只能对数据条带进行 GC。如果你周围有一堆旧消息,它永远无法从磁盘中收集这些数据条带。

这可能发生在任何队列中有真正的旧消息。尽快消费或丢弃消息。紧急处理 DLQ,或者根本不使用它;在您的配置中禁用 DLQ 并让错误消失,让业务流程在其他地方查找错误。

在某些情况下,将消息移动到另一个队列并返回可以让 KahaDB 进行一些压缩,但是这里应用了很多星号,因为它必须在两个单独的事务中并且存在风险。最好的建议仍然是:经纪人不是存储设备。

如果 DLQ 不重要,例如:您可以无追索权地删除消息,那么您很可能不需要持久交付,因为每条消息并不是那么重要。如果不需要 100% 的可靠性,您可以关闭持久传递,然后消息将不会一定被写出到磁盘。这也可能会改善这种情况,但关闭持久交付取决于您的业务需求以及您在堆栈中使用消息传递的方式。

最后,如果你所有的队列都是空的,你可以关闭你的代理并清空数据目录,然后重新启动它。这将导致 KahaDB 重新开始。不过,这是一个短期修复,因为问题的根源(不处理 DLQ)可能仍然存在。

【讨论】:

    猜你喜欢
    • 2023-03-12
    • 1970-01-01
    • 2015-08-12
    • 1970-01-01
    • 2011-03-19
    • 2018-02-19
    • 2021-03-13
    • 2011-06-21
    • 2012-08-01
    相关资源
    最近更新 更多