报错背景:
CDH断电重启后,cloudera-scm-server启动报错,
cloudera-scm-server 已死,但 pid 文件仍存
由于没有成熟的解决方案,于是我就重新安装了MySQL数据库,
并且重新赋予了各种权限。
这个时候cloudera-scm-server可以启动成功,cloudera-scm-agent也可以启动成功。
报错现象:
此时cloudera-scm-server和cloudera-scm-agent都能启动成功,
但是jps一下,没有服务
明明启动了agent,但是下面的服务都起不来。
报错原因:
这个时候就要看一下日志文件cloudera-scm-agent.log
tail -n 100 /opt/cm-5.15.1/log/cloudera-scm-agent/cloudera-scm-agent.log
看到一行报错
[root@master opt]# tail -n 100 /opt/cm-5.15.1/log/cloudera-scm-agent/cloudera-scm-agent.log [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '4754', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml httpfs httpfs [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'httpfs:httpfs', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.dist/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/sqoop2/tomcat-conf.mr1/conf/server.xml sqoop2 sqoop [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'sqoop2:sqoop', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/sqoop2/tomcat-conf.mr1/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/sqoop2/tomcat-conf.mr1/conf/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/sqoop2/tomcat-conf.mr1/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop hue hue [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop 0755 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0755', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml httpfs httpfs [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'httpfs:httpfs', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/hadoop-httpfs/tomcat-conf.https/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https/conf/ssl/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.https.mr1/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/solr/tomcat-conf.https/conf/server.xml solr solr [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'solr:solr', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/solr/tomcat-conf.https/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/solr/tomcat-conf.https/conf/server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/solr/tomcat-conf.https/conf/server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chown: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml oozie oozie [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chown', 'oozie:oozie', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml 0750 [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['chmod', '0750', u'/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/etc/oozie/tomcat-conf.http.mr1/conf/ssl/ssl-server.xml'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Ensuring users/groups exist for new parcel KAFKA-3.1.1-1.3.1.1.p0.2. [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'kafka'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'kafka'] [29/Apr/2019 10:35:14 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'kafka', '-K', 'UMASK=022', '--home', '/var/lib/kafka', '--comment', 'Kafka', '--shell', '/sbin/nologin', 'kafka'] [29/Apr/2019 10:35:15 +0000] 2754 MainThread parcel INFO Ensuring correct file permissions for new parcel KAFKA-3.1.1-1.3.1.1.p0.2. [29/Apr/2019 10:35:15 +0000] 2754 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache [29/Apr/2019 10:35:16 +0000] 2754 MainThread client_configs INFO Loaded client config: <hbase-conf,/etc/hbase/conf.cloudera.hbase> <90,8014743985700882227> [29/Apr/2019 10:35:16 +0000] 2754 MainThread client_configs INFO Loaded client config: <hadoop-conf,/etc/hadoop/conf.cloudera.yarn> <92,-2128374979962437415> [29/Apr/2019 10:35:16 +0000] 2754 MainThread client_configs INFO Loaded client config: <kafka-conf,/etc/kafka/conf.cloudera.kafka> <50,-352371731053066898> [29/Apr/2019 10:35:16 +0000] 2754 MainThread client_configs INFO Loaded client config: <hive-conf,/etc/hive/conf.cloudera.hive> <90,-3132036662857147630> [29/Apr/2019 10:35:16 +0000] 2754 MainThread client_configs INFO Loaded client config: <hadoop-conf,/etc/hadoop/conf.cloudera.hdfs> <90,-8675592532172053281> [29/Apr/2019 10:35:16 +0000] 2754 MainThread agent ERROR Error, CM server guid updated, expected f4ec230a-2f55-460f-b132-cb45a16c3a1b, received 4760feb5-8e8c-416c-8337-2910fef890a0 [29/Apr/2019 10:35:18 +0000] 2754 MainThread agent INFO CM server guid: f4ec230a-2f55-460f-b132-cb45a16c3a1b [29/Apr/2019 10:35:18 +0000] 2754 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels [29/Apr/2019 10:35:18 +0000] 2754 MainThread agent WARNING Expected user root for /opt/cloudera/parcels but was cloudera-scm [29/Apr/2019 10:35:18 +0000] 2754 MainThread agent WARNING Expected group root for /opt/cloudera/parcels but was cloudera-scm [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Agent does create users/groups and apply file permissions [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Loading parcel manifest for: CDH-5.15.1-1.cdh5.15.1.p0.4 [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Loading parcel manifest for: KAFKA-3.1.1-1.3.1.1.p0.2 [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Ensuring users/groups exist for new parcel CDH-5.15.1-1.cdh5.15.1.p0.4. [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hadoop'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'flume'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hdfs'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'solr'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'zookeeper'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'llama'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'httpfs'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'mapred'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'sqoop'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'yarn'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'kms'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hive'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'sqoop2'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'oozie'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'kudu'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'hbase'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'sentry'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'impala'] [29/Apr/2019 10:35:18 +0000] 2754 MainThread parcel INFO Executing command ['/usr/sbin/groupadd', '-r', 'spark']