在 Apache hive 上执行分析查询（例如 select count(*) ）时出错

Question

当我像 select * from table3 limit 100; 对配置单元执行简单查询时，结果还可以。 但是当我执行 select count(*) from table3 ; 我遇到以下错误：

 0: hiv> select count(*) from table3;
INFO  : Compiling command(queryId=hive_20200329113344_1afc6405-bf94-4c96-a7a3-50d97a45f7bb): select count(*) from table3
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20200329113344_1afc6405-bf94-4c96-a7a3-50d97a45f7bb); Time taken: 0.24 seconds
INFO  : Executing command(queryId=hive_20200329113344_1afc6405-bf94-4c96-a7a3-50d97a45f7bb): select count(*) from table3
INFO  : Query ID = hive_20200329113344_1afc6405-bf94-4c96-a7a3-50d97a45f7bb
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
WARN  : The session: sessionId=d229f221-9c07-449b-bf8d-651c6392bb70, queueName=null, user=hive, doAs=true, isOpen=false, isDefault=false has not been opened
INFO  : Subscribed to counters: [] for queryId: hive_20200329113344_1afc6405-bf94-4c96-a7a3-50d97a45f7bb
INFO  : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1582632679991_0006 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 1 applications, cannot accept submission of application: application_1582632679991_0006
        at org.apache.tez.client.TezClient.start(TezClient.java:407) ~[tez-api-0.9.1.3.0.1.0-187.jar:0.9.1.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:516) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:451) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.openInternal(TezSessionPoolSession.java:124) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:373) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:372) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:199) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2701) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2372) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2048) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1746) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1740) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) ~[hive-exec-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:318) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_171]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_171]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.1.1.3.0.1.0-187.jar:?]
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:331) ~[hive-service-3.1.0.3.0.1.0-187.jar:3.1.0.3.0.1.0-187]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_171]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_171]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_171]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1582632679991_0006 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 1 applications, cannot accept submission of application: application_1582632679991_0006
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:304) ~[hadoop-yarn-client-3.1.1.3.0.1.0-187.jar:?]
        at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77) ~[tez-api-0.9.1.3.0.1.0-187.jar:0.9.1.3.0.1.0-187]
        at org.apache.tez.client.TezClient.start(TezClient.java:402) ~[tez-api-0.9.1.3.0.1.0-187.jar:0.9.1.3.0.1.0-187]
        ... 28 more
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
INFO  : Completed executing command(queryId=hive_20200329113344_1afc6405-bf94-4c96-a7a3-50d97a45f7bb); Time taken: 0.422 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)

这是我在该应用程序的纱线 ResourceManager UI >> 应用程序中看到的错误

org.apache.hadoop.security.AccessControlException: Queue root.default already has 1 applications, cannot accept submission of application: application_1582632679991_0006

Answer 1

这主要是由于yarn.scheduler.capacity.maximum-applications 限制被击中。

杀死不必要的作业并在yarn-site.xml 配置中增加该属性。