在 AWS 上流畅安装

Question

实际上，我们计划在节点服务器上使用 fluentd 来获取日志并将这些日志转发到我们的日志收集器（主 Fluentd 服务器），然后我们可以将日志传输到 s3 存储桶，现在我们可以将日志写入我的 s3存储桶，但在调试模式下重新启动 td-agent 时出现错误。

流畅的节点配置：-

root@ip-172-31-32-115:/var/log/td-agent# cat /etc/td-agent/td-agent.conf
#Apache Access logs
<source>
  @type tail
  format apache2
  path /var/log/apache2/access.log
  pos_file /var/log/td-agent/apache2.access.pos
  tag apache.access
</source>
#Log forwardin
<match apache.access>
  @type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 1s
  phi_threshold 16
  hard_timeout 60s
#primary host

  <server>
    host xx.xx.xx.xx
    port 24224
  </server>
  flush_interval 60s
</match>

日志转发器日志

root@ip-172-31-34-201:/home/ubuntu# td-agent -v 
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:457:read_config: reading config file path="/etc/td-agent/td-agent.conf"
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:323:supervise: starting fluentd-0.12.20
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-mongo' version '0.7.12'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-s3' version '0.6.5'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td' version '0.10.28'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluentd' version '0.12.20'
2016-06-06 18:21:44 +0000 [info]: fluent/agent.rb:125:add_match: adding match pattern="apache2.." type="forward"
2016-06-06 18:21:44 +0000 [info]: plugin/out_forward.rb:141:block in configure: adding forwarding server '172.31.33.196:24224' host="172.31.33.196" port=24224 weight=60 plugin_id="object:11e9668"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail" plugin/out_forward.rb:232:rebuild_weight_array: rebuilding weight array lost_weight=0
2016-06-06 18:21:44 +0000 [info]: plugin/in_tail.rb:499:initialize: following tail of /var/log/apache2/access.log

日志聚合器配置：-

#Input
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>
# S3 config
<match s3.*.*>
    @type s3

    aws_key_id AKIERPE6JxxxKXSG3GWSA
    aws_sec_key 31KnCOJdQrxxxrXEw0dYY5RqV9zegTedfrere25F1Hsvw
    s3_bucket rana.sahu
    s3_region us-west-2
    path logs/
    buffer_type           file
    buffer_path           /var/log/td-agent/s3/buffer/td
    buffer_chunk_limit    2m
    buffer_queue_limit    4098
    retry_wait            1s
    max_retry_wait        5s
    disable_retry_limit   true


    time_slice_format %Y%m%d%H
    time_slice_wait 10m
    utc

</match>

卡在这个东西上

当我启动 td-agent -v 我得到这个错误 2016-06-07 07:11:00 +0000 [info]: plugin/in_forward.rb:81:listen: 在 0.0.0.0:24224 上监听 fluent socket 2016-06-07 07:11:00 +0000 [错误]: fluent/engine.rb:196:rescue in run: 意外错误 error_class=Errno::EADDRINUSE 错误=# 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool .io-1.4.2/lib/cool.io/server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:innew' 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool .io-1.4.2/lib/cool.io/server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:82:innew' 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20/lib/fluent/plugin/in_forward.rb:82:in listen' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:51:instart' 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20/lib/fluent/root_agent.rb:111:in block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:110:ineach' 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20/lib/fluent/root_agent.rb:110:in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:225:instart' 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20/lib/fluent/engine.rb:175:in run' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:597:inrun_engine' 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20/lib/fluent/supervisor.rb:148:in block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:incall' 2016-06-07 07:11:00 +0000 [错误]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20/lib/fluent/supervisor.rb:352:in `main_process'

如果您对此有任何解决方案，请告诉我。

Answer 1

我得到了答案，通过这些配置，我们可以将 apache2 日志转发到日志收集器并将它们存储到 s3 存储桶中。

谢谢大家。