【问题标题】:Cannot use supervisor with Apache Kafka无法将主管与 Apache Kafka 一起使用
【发布时间】:2017-01-23 22:15:21
【问题描述】:

我有一台安装了 Apache Kafka 的 Ubuntu 16.04 机器。目前,我可以通过使用具有以下内容的start_kafka.sh 脚本使其完美运行:

JMX_PORT=17264 KAFKA_HEAP_OPTS="-Xms1024M -Xmx3072M" /home/kafka/kafka_2.11-0.10.1.0/bin/kafka-server-start.sh -daemon /home/kafka/kafka_2.11-0.10.1.0/config/server.properties

现在,我想使用supervisor 自动重启进程,如果它失败并在重启机器后立即启动。问题是我不能让supervisor 启动Kafka。

我使用pip安装了supervisor,并将这个配置文件放在/etc/supervisord.conf

; Supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[program:kafka]
command=/home/kafka/kafka_2.11-0.10.1.0/start_kafka.sh ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
startsecs=10                   ; # of secs prog must stay up to be running (def. 1)
startretries=3                ; max # of serial start failures when starting (default 3)
;autorestart=unexpected        ; when to restart if exited after running (def: unexpected)
;exitcodes=0,2                 ; 'expected' exit codes used with autorestart (default 0,2)
stopsignal=TERM               ; signal used to kill process (default TERM)
stopwaitsecs=180               ; max num secs to wait b4 SIGKILL (default 10)
stdout_logfile=NONE        ; stdout log path, NONE for none; default AUTO
;environment=A="1",B="2"       ; process environment additions (def no adds)

当我尝试启动 Kafka 时,出现以下错误:

# supervisorctl start kafka
kafka: ERROR (spawn error)

主管日志(/tmp/supervisord.log)包含以下内容:

2017-01-23 22:10:24,532 INFO spawned: 'kafka' with pid 21311
2017-01-23 22:10:24,536 INFO exited: kafka (exit status 127; not expected)
2017-01-23 22:10:25,542 INFO spawned: 'kafka' with pid 21312
2017-01-23 22:10:25,559 INFO exited: kafka (exit status 127; not expected)
2017-01-23 22:10:27,562 INFO spawned: 'kafka' with pid 21313
2017-01-23 22:10:27,567 INFO exited: kafka (exit status 127; not expected)
2017-01-23 22:10:30,571 INFO spawned: 'kafka' with pid 21314
2017-01-23 22:10:30,576 INFO exited: kafka (exit status 127; not expected)
2017-01-23 22:10:31,578 INFO gave up: kafka entered FATAL state, too many start retries too quickly

必须说我已经尝试删除start_kafka.sh 中的-daemon 标志以与supervisor 一起使用,但没有成功。

有人知道发生了什么吗?

【问题讨论】:

  • 'command=/home/kafka/kafka_2.11-0.10.1.0/start_kafka.sh' ?为什么它不包含'config/server.properties'?

标签: ubuntu apache-kafka ubuntu-16.04 supervisord


【解决方案1】:

以下主管配置文件适用于我,取自https://github.com/miguno/wirbelsturm 通过https://github.com/miguno/puppet-kafka。主要区别在于它使用kafka-run-class.sh 而不是kafka-server-start.sh

请注意,您需要更新各种路径,使其与您的设置相匹配,例如您必须将/opt/kafka/bin/kafka-run-class.sh 更改为/home/kafka/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh

[program:kafka-broker]
command=/opt/kafka/bin/kafka-run-class.sh kafka.Kafka /opt/kafka/config/server.properties
numprocs=1
numprocs_start=0
priority=999
autostart=true
autorestart=true
startsecs=10
startretries=999
exitcodes=0,2
stopsignal=INT
stopwaitsecs=120
stopasgroup=true
directory=/
user=kafka
redirect_stderr=false
stdout_logfile=/var/log/supervisor/kafka-broker/kafka-broker.out
stdout_logfile_maxbytes=20MB
stdout_logfile_backups=5
stderr_logfile=/var/log/supervisor/kafka-broker/kafka-broker.err
stderr_logfile_maxbytes=20MB
stderr_logfile_backups=10
environment=JMX_PORT=9999,KAFKA_GC_LOG_OPTS="-Xloggc:/var/log/kafka/daemon-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps",KAFKA_HEAP_OPTS="-Xms512M -Xmx512M -XX:NewSize=200m -XX:MaxNewSize=200m",KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false",KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true",KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/opt/kafka/config/log4j.properties",KAFKA_OPTS="-XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintTenuringDistribution"

【讨论】:

    【解决方案2】:

    我终于设法让主管使用 Kafka 进行了两项更改:

    • 部署不带 -daemon 标志的 Kafka,因为主管需要非守护进程来管理
    • 在主管配置文件中明确定义 Java 路径

    这是工作配置:

    start_kafka.sh

    JMX_PORT=17264 KAFKA_HEAP_OPTS="-Xms1024M -Xmx3072M" /home/kafka/kafka_2.11-0.10.1.0/bin/kafka-server-start.sh /home/kafka/kafka_2.11-0.10.1.0/config/server.properties
    

    supervisord.conf

    [unix_http_server]
    file=/var/run/supervisor.sock   ; (the path to the socket file)
    chmod=0700                       ; sockef file mode (default 0700)
    
    [supervisord]
    logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
    pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
    childlogdir=/var/log/supervisor            ; ('AUTO' child log dir, default $TEMP)
    
    ; the below section must remain in the config file for RPC
    ; (supervisorctl/web interface) to work, additional interfaces may be
    ; added by defining them in separate rpcinterface: sections
    [rpcinterface:supervisor]
    supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
    
    [supervisorctl]
    serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket
    
    ; The [include] section can just contain the "files" setting.  This
    ; setting can list multiple files (separated by whitespace or
    ; newlines).  It can also contain wildcards.  The filenames are
    ; interpreted as relative to this file.  Included files *cannot*
    ; include files themselves.
    
    [include]
    files = /etc/supervisor/conf.d/*.conf
    
    [program:kafka]
    command=/home/kafka/kafka_2.11-0.10.1.0/start_kafka.sh
    directory=/home/kafka/kafka_2.11-0.10.1.0
    user=root
    autostart=true
    autorestart=true
    stdout_logfile=/var/log/kafka/stdout.log
    stderr_logfile=/var/log/kafka/stderr.log
    environment = JAVA_HOME=/usr/lib/jvm/java-8-oracle
    

    【讨论】:

      猜你喜欢
      • 2019-10-31
      • 1970-01-01
      • 2018-03-12
      • 2018-11-23
      • 2020-01-03
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2015-08-12
      相关资源
      最近更新 更多