【问题标题】:Flink 1.5.1 on yarn纱线上的 Flink 1.5.1
【发布时间】:2018-08-15 11:17:30
【问题描述】:

我正在使用 flink 1.5.1 ,一个集群一个作业

我使用 CLI 向 yarn 提交了一个作业,然后取消它,有一些我无法理解的错误

:~/flink-1.5.1/bin$ ./flink run -d -m yarn-cluster -ynm lz_test_alone1   -ytm 4096 -yjm 1024  -c Demo  /home/test/flink1.5_demo-1.1-SNAPSHOT.jar
2018-08-10 15:05:28,390 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-sloth.
2018-08-10 15:05:28,390 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-sloth.
2018-08-10 15:05:28,937 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2018-08-10 15:05:28,937 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2018-08-10 15:05:29,324 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
2018-08-10 15:05:29,818 WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory       - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2018-08-10 15:05:29,824 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/home/flink-1.5.1/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2018-08-10 15:05:31,087 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1526888270443_0071
2018-08-10 15:05:31,121 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1526888270443_0071
2018-08-10 15:05:31,121 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2018-08-10 15:05:31,126 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2018-08-10 15:05:35,644 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
2018-08-10 15:05:35,644 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The Flink YARN client has been started in detached mode. In order to stop Flink on YARN, use the following command or a YARN web interface to stop it:
yarn application -kill application_1526888270443_0071
Please also note that the temporary files of the YARN session in the home directory will not be removed.
Job has been submitted with JobID 4c4f9564adc94cd38d91d46660ade0d9

:~/flink-1.5.1/bin$ ./flink cancel 4c4f9564adc94cd38d91d46660ade0d9
2018-08-10 15:05:55,484 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-sloth.
2018-08-10 15:05:55,484 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-sloth.
Cancelling job 4c4f9564adc94cd38d91d46660ade0d9.
2018-08-10 15:05:55,837 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 3
2018-08-10 15:05:55,837 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 3
YARN properties set default parallelism to 3
2018-08-10 15:05:56,026 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2018-08-10 15:05:56,026 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2018-08-10 15:05:56,116 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Found application JobManager host name 'cluster23.org' and port '57813' from supplied application id 'application_1526888270443_0067'
2018-08-10 15:05:56,707 WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory       - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

当我使用返回 jobID 和 applicationId (application_1526888270443_0071) 的“flink run”时;

然后我使用'flink cancel',日志显示另一个applicationId(application_1526888270443_0067),所以出错了。

有没有人可以帮助我?

【问题讨论】:

    标签: hadoop-yarn apache-flink


    【解决方案1】:
    ./flink cancel  
    

    用于纱线工作需要 -yid 这个脚本应该是:

    ./flink cancel  4c4f9564adc94cd38d91d46660ade0d9 -yid application_1526888270443_0071
    

    这行得通

    【讨论】:

      猜你喜欢
      • 2020-09-07
      • 1970-01-01
      • 1970-01-01
      • 2020-10-07
      • 2021-01-31
      • 2019-12-20
      • 2018-11-24
      • 2021-05-27
      • 2019-12-31
      相关资源
      最近更新 更多