【问题标题】:Flume agent unable to deliver eventFlume 代理无法传递事件
【发布时间】:2013-12-23 14:21:57
【问题描述】:

我有 3 个代理。 1 使用内存通道在 windows 上运行,其他 2 在 Linux 上使用文件通道从 windows 代理获取数据并放入 Hbase。

谁能指出为什么会发生以下错误以及阻止它的步骤是什么?

2013-12-23 14:50:15,290 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.sink.AvroSink.destroyConnection(AvroSink.java:199)] Avro sinksink1 closing avro client: NettyAvroRpcClient { host: 192.168.101.232, port: 3001 }
2013-12-23 14:50:15,290 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event.
Exception follows.org.apache.flume.EventDeliveryException: Failed to send events
        at org.apache.flume.sink.AvroSink.process(AvroSink.java:325)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: 192.168.101.232, port: 3001 }: Failed to send batch
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:236)
        at org.apache.flume.sink.AvroSink.process(AvroSink.java:309)        ... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: 192.168.101.232, port: 3001 }: Handshake timed out after 20000ms
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:280)
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:224)    ... 4 more
Caused by: java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:278)

下面是我的windows配置文件

a1.sources = source1
a1.channels = channel1 channel2
a1.sinks = sink1 sink2

a1.sources.source1.handler = com.flume.handler.DynamicJSONHandler
a1.sources.source1.type = org.apache.flume.source.http.HTTPSource
a1.sources.source1.bind = 192.168.101.29
a1.sources.source1.port = 2001

a1.channels.channel1.type = org.apache.flume.channel.MemoryChannel
a1.channels.channel1.capacity = 1000
a1.channels.channel1.transactionCapacity = 1000

a1.sinks.sink1.type = org.apache.flume.sink.AvroSink
a1.sinks.sink1.hostname = 192.168.101.232
a1.sinks.sink1.port = 3001

a1.channels.channel2.type = org.apache.flume.channel.MemoryChannel
a1.channels.channel2.capacity = 1000
a1.channels.channel2.transactionCapacity = 1000

a1.sinks.sink2.type = org.apache.flume.sink.AvroSink
a1.sinks.sink2.hostname = 192.168.101.233
a1.sinks.sink2.port = 3001

a1.sources.source1.channels = channel1 channel2
a1.sinks.sink1.channel = channel1
a1.sinks.sink2.channel = channel2

【问题讨论】:

    标签: java linux apache hbase flume


    【解决方案1】:

    AvroSink 旨在将数据写入另一个具有 Avro Source 侦听该特定源的水槽代理。 Avro Sink 启动与它所连接的源的握手。但看起来握手在 20 秒后超时 - 因此您的网络很可能正面临严重的延迟问题。您在运行源代码的代理上是否发现任何问题?

    【讨论】:

    • 同时我的源代理没有收到任何问题。
    • 但是当我减少 Avro Sink 的负载时,错误消失了。感谢回复
    • @Shah 你如何减少负载,你能解释一下吗?
    猜你喜欢
    • 2019-10-30
    • 2018-10-08
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多