kafka-python 引发 kafka.errors.ConsumerFetchSizeTooSmall答案

【问题标题】：kafka-python raise kafka.errors.ConsumerFetchSizeTooSmallkafka-python 引发 kafka.errors.ConsumerFetchSizeTooSmall
【发布时间】：2016-09-13 20:58:43
【问题描述】：

我正在 python 2.7 中编写一个简单的代码，它正在使用来自 apache kafka 主题的消息。代码如下：

from kafka import SimpleConsumer,KafkaClient
group = "my_group_test"
client = KafkaClient('localhost:9092')
cons = SimpleConsumer(client, group, "my_topic")
messages = cons.get_messages(count=1000,block=False)

但是引发了这个异常：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/simple.py", line 285, in get_messages
    update_offset=False)
  File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/simple.py", line 320, in _get_message
    self._fetch()
  File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/simple.py", line 425, in _fetch
    raise ConsumerFetchSizeTooSmall()
kafka.errors.ConsumerFetchSizeTooSmall

如何修改此参数 (ConsumerFetchSize) 以使此代码正常工作？

【问题讨论】：

标签： python python-2.7 apache-kafka kafka-python

【解决方案1】：

我找到了解决方案，在 SimpleConsumer 中使用参数max_buffer_size。

工作代码是：

#the size is adherent with my need but is totally arbitrary
cons = SimpleConsumer(client, group, "my_topic",max_buffer_size=9932768)

【讨论】：