【问题标题】:Alternative methods of computing real-time event frequency计算实时事件频率的替代方法
【发布时间】:2020-02-12 14:10:33
【问题描述】:

我正在使用一个有大量外部事件报告的应用程序。经常使用的指标之一是作为时间函数的事件率。例如,测量某些外部异步传感器的采样率。

目前我计算此类事件频率的方式是保留事件时间戳队列。当事件发生时,我们将当前时间戳推送到队列中,然后弹出直到最旧的时间戳小于预定义的年龄。然后,事件频率与队列的大小成正比。在伪代码中,该方法通常看起来像这样:

def on_event():
    var now = current_time()
    time_queue.push(now)

    while((now - time_queue.front()) > QUEUE_DEPTH_SECONDS):
        time_queue.pop()

    frequency = time_queue.size() / QUEUE_DEPTH_SECONDS

现在这种方法显然不是最优的:

  1. 内存需求和计算时间与事件率成正比。
  2. 必须根据预期的数据速率手动调整队列持续时间,以调整低频性能与内存要求。
  3. 频率测量的响应时间也取决于队列持续时间。较长的持续时间会降低计算的响应时间。
  4. 仅在发生新事件时更新频率。如果事件停止发生,则频率测量将保持在接收到最后一个事件时计算的值。

我很好奇是否有任何替代算法可用于计算事件发生率,以及它们在计算复杂性、空间要求、响应时间等方面的权衡。

【问题讨论】:

    标签: algorithm frequency counting frequency-analysis


    【解决方案1】:

    我已经实现了类似的东西来计算气流中粒子的速率/浓度。它们来自随机事件(可能是泊松分布),我想知道每次粒子的平均速率。我的方法如下(取自我的代码的文档字符串):

    事件时间戳被放置在一个固定的最大大小的缓冲区中。 每当请求浓度估计时,所有时间戳 过滤掉设置的最大年龄。如果剩余数量 据报道,最近的时间戳低于某个阈值 浓度为零。否则,计算浓度 剩余事件数除以时间差 在缓冲区中最早的时间戳和现在之间。

    我将附上下面的 Python 实现以供参考。它是一个更大项目的一部分,所以我不得不稍微修改它以摆脱对外部代码的引用:

    particle_concentration_estimator_params.py

    #!/usr/bin/env python3
    
    """This module implements a definition of parameters for the particle
    concentration estimator.
    """
    
    __author__    = "bup"
    __email__     = "bup@swisens.ch"
    __copyright__ = "Copyright 2021, Swisens AG"
    __license__   = "GNU General Public License Version 3 (GPLv3)"
    
    __all__ = ['ParticleConcentrationEstimatorParams']
    
    
    import dataclasses
    
    
    @dataclasses.dataclass
    class ParticleConcentrationEstimatorParams:
        """This provides storage for the parameters used for the particle
        concentration estimator.
        """
    
        timestamp_buffer_max_size: int
        """Maximum size of the buffer that is used to keep track of event
        timestamps. The size of this buffer mainly affects the filtering
        of the reported data.
    
        Unit: - (count)
        """
    
        timestamp_buffer_max_age: float
        """Maximum age of events in the timestamp buffer which are
        considered for the concentration calculation. This value is a
        tradeoff between a smooth filtered value and the dynamic response
        to a changed concentration.
    
        Unit: s
        """
    
        min_number_of_timestamps: int
        """Minimum number of timestamps to use for the concentration
        estimation. If less timestamps are available, the concentration is
        reported as zero.
    
        Unit: - (count)
        """
    

    particle_concentration_estimator.py

    #!/usr/bin/env python3
    
    """This module implements the particle concentration estimation.
    """
    
    __author__    = "bup"
    __email__     = "bup@swisens.ch"
    __copyright__ = "Copyright 2021, Swisens AG"
    __license__   = "GNU General Public License Version 3 (GPLv3)"
    
    __all__ = ['ParticleConcentrationEstimator']
    
    
    import logging
    import time
    from typing import Optional
    
    import numpy as np
    
    from .particle_concentration_estimator_params import ParticleConcentrationEstimatorParams
    
    
    logger = logging.getLogger(__name__)
    
    
    class ParticleConcentrationEstimator:
        """An object of this class implements the Poleno particle
        concentration estimator. Particle concentration is basically just
        a number of particles per time unit. But since the particle events
        arrive irregularly, there are various ways to filter the result, to
        avoid too much noise especially when the concentration is low. This
        class implements the following approach:
    
        Event timestamps are placed in a buffer of a fixed maximum size.
        Whenever a concentration estimate is requested, all timestamps up to
        a set maximum age are filtered out. If the remaining number of
        recent timestamps is below some threshold, the reported
        concentration is zero. Otherwise, the concentration is calculated
        as the number of remaining events divided by the time difference
        between the oldest timestamp in the buffer and now.
        """
    
        def __init__(self, params: ParticleConcentrationEstimatorParams):
            """Initializes the object with no events.
    
            Args:
                est_params: Initialized PolenoParams object which includes
                    information describing the estimator's behaviour.
            """
            self.params = params
            """Stored params for the object."""
            n_rows = self.params.timestamp_buffer_max_size
            self._rb = np.full((n_rows, 2), -1e12)
            self._rb_wp = 0
            self._num_timestamps = 0
            self._concentration_value = 0.0
            self._concentration_value_no_mult = 0.0
    
        def tick(self, now: float) -> float:
            """Recalculates the current concentration value.
            
            Args:
                now: Current timestamp to use to filter out old entries
                    in the buffer.
    
            Returns:
                The updated concentration value, which is also returned
                using the concentration attribute.
            """
            min_ts = now - self.params.timestamp_buffer_max_age
            min_num = self.params.min_number_of_timestamps
            used_rows = self._rb[:, 0] >= min_ts
            filt_ts = self._rb[used_rows]
            num_ts = round(np.sum(filt_ts[:, 1]))
            self._num_timestamps = num_ts
            num_ts_no_mult = round(np.sum(used_rows))
            if num_ts < min_num:
                self._concentration_value = 0.0
                self._concentration_value_no_mult = 0.0
            else:
                t_diff = now - np.min(filt_ts[:, 0])
                if t_diff >= 1e-3:
                    # Do not change the reported value if all events in the
                    # buffer have the same timestamp.
                    self._concentration_value = num_ts / t_diff
                    self._concentration_value_no_mult = num_ts_no_mult / t_diff
            return self._concentration_value
    
        def got_timestamp(self,
                          ts: Optional[float] = None,
                          multiplier: float = 1.0) -> None:
            """Passes in the most recent event timestamp. Timestamps need
            not be ordered.
    
            Calling this method does not immediately update the
            concentration value, this is deferred to the tick() method.
    
            Args:
                ts: Event timestamp to use. If None, the current time is
                    used.
                multiplier: Optional multiplier, which makes ts to be
                    counted that many times.
            """
            if ts is None:
                ts = time.time()
            self._rb[self._rb_wp] = (ts, float(multiplier))
            self._rb_wp = (self._rb_wp + 1) % self._rb.shape[0]
            self._num_timestamps += round(multiplier)
    
        @property
        def concentration(self) -> float:
            """The calculated concentration value.
    
            Unit: 1/s
            """
            return self._concentration_value
    
        @property
        def concentration_no_mult(self) -> float:
            """The calculated concentration value without taking into
            account the timestamp multipliers, i.e. as if all timestamps
            were given with the default multiplier of 1.
    
            Unit: 1/s
            """
            return self._concentration_value_no_mult
    
        @property
        def num_timestamps(self) -> int:
            """Returns the number of timestamps which currently are in
            the internal buffer.
            """
            return self._num_timestamps
    

    【讨论】:

      【解决方案2】:

      https://en.wikipedia.org/wiki/Exponential_smoothing 非常高效,只使用少量有限的内存。您可以尝试对到达间隔时间进行指数平滑。在检索平滑的到达间隔时间时,您可以查看最后一个事件的时间,如果它大于平滑的到达间隔时间,则将其混合。

      这已经足够不同了,实际上我会从收集当前使用的时间戳样本开始,这样我就可以用它来离线测试这个或其他方案的结果。

      【讨论】:

        【解决方案3】:

        另一种选择是以恒定速率触发的本地计时器(例如每秒一次):

        • 当外部事件发生时,它会增加一个计数器。
        • 当本地定时器触发时,计数器值被添加到队列中,并且计数被重置为零。

        以下是该方法与您的方法的比较:

        1. 内存要求和计算时间与外部事件率无关。它由您控制的本地计时器速率确定。
        2. 队列大小取决于您要进行多少平均。队列大小为 1(即无队列),定时器速率为每秒一次,会导致每秒读取原始事件,没有平均值。队列越大,您获得的平均次数就越多。
        3. 响应时间由所需的平均量决定。更多的平均会导致更慢的响应时间。
        4. 无论是否发生外部事件,都会以本地计时器触发的频率更新频率。

        【讨论】:

          猜你喜欢
          • 1970-01-01
          • 2017-07-22
          • 1970-01-01
          • 2012-02-01
          • 1970-01-01
          • 1970-01-01
          • 1970-01-01
          • 2013-10-09
          • 1970-01-01
          相关资源
          最近更新 更多