一个月前我在 Go 中做了同样的实现。我的假设如下:
- 如果队列中有消息,应用程序会非常快速地消耗它们(2 条消息之间的时间少于 100 毫秒)。
- 如果队列为空(我的应用程序已完成消费所有消息),则可以收到新消息,但慢于 100 毫秒
因此,我实现了这个:
* 每次收到消息,
* 我暂停 100ms 超时
* 我处理并确认消息
* 我将 100 毫秒超时重置为 0
* 如果触发了 100 毫秒超时,我将终止我的拉取订阅
在我的用例中,我安排每 10 分钟处理一次。所以,我在 9 分 30 秒设置了一个全局超时来完成处理,让新的应用实例继续处理
只是一个棘手的事情:对于第一条消息,将超时设置为 2 秒。实际上,由于连接建立,第一条消息消息需要更长的时间才能到达。因此,在您初始化超时时设置一个标志“是否是第一条消息”。
如果我的 Go 代码可以帮助您实现,我可以分享它。
编辑
这是我关于消息处理的 Go 代码
func (pubSubService *pubSubService) Received() (msgArray []*pubsub.Message, err error) {
ctx := context.Background()
cctx, cancel := context.WithCancel(ctx)
// Connect to PubSub
client, err := pubsub.NewClient(cctx, pubSubService.projectId)
if err != nil {
log.Fatalf("Impossible to connect to pubsub client for project %s", pubSubService.projectId)
}
// Put all the message in a array. It will be processed at the end (stored to BQ, as is)
msgArray = []*pubsub.Message{}
// Channel to receive messages
var receivedMessage = make(chan *pubsub.Message)
// Handler to receive message (through the channel) or cancel the the context if the timeout is reached
go func() {
//Initial timeout because the first received is longer than this others.
timeOut := time.Duration(3000)
for {
select {
case msg := <-receivedMessage:
//After the first receive, the timeout is changed
timeOut = pubSubService.waitTimeOutInMillis // Environment variable = 200
msgArray = append(msgArray, msg)
case <-time.After(timeOut * time.Millisecond):
log.Debug("Cancel by timeout")
cancel()
return
}
}
}()
// Global timeout
go func(){
timeOut = pubSubService.globalWaitTimeOutInMillis // Environment variable = 750
time.Sleep(timeOut * time.Second):
log.Debug("Cancel by global timeout")
cancel()
return
}
// Connect to the subscription and pull it util the channel is canceled
sub := client.Subscription(pubSubService.subscriptionName)
err = sub.Receive(cctx, func(ctx context.Context, msg *pubsub.Message) {
receivedMessage <- msg
msg.Ack()
})
}