环境:刚搭好的CDH5.13.1,集群5台机器,HIVE底层MR。
hive命令行输入select count(*)或者insert into 都会报以下错误:

日志描述

hive insert into count(*)报错
以下是application日志:http://hadoop01:8088
hive insert into count(*)报错
Jobhistory日志:http://hadoop01:19888/jobhistory
hive insert into count(*)报错
查看yarn log(命令行输入如下):
#yarn logs -applicationId application_1552964743608_0001 > logs_20190319.txt

更改过的配置:

发现报错后,上网各种百度谷歌,改了很多配置,大部分都失败了,但是中间改过一次mapred-site.xml的address,后来又改回去了反而成功了,所以估计是中间某步改成功了,还好改过的基本都记录了,主要修改过以下几个配置文件,特此记录一下,也供大家参考。
1.yarn-site.xml
2.mapred-site.xml
3./etc/profile
4.hive-site.xml

1.改yarn-site.xml
配置文件:
#find / -name yarn-site.xml
查到有以下五个比较像,但不确定是哪个,故各项都同时修改以下五个:
1)/etc/hadoop/conf.cloudera.yarn/yarn-site.xml
2)/etc/hive/conf.cloudera.hive/yarn-site.xml
3)/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/yarn-site.xml
4)/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/hadoop/conf.empty/yarn-site.xml
5)/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/hadoop/conf.pseudo/yarn-site.xml
修改参数:
1)增加yarn.nodemanager.resource.memory-mb,值改为:14510
2)yarn.application.classpath,值改为:
hive insert into count(*)报错
3)yarn.scheduler.minimum-allocation-mb,值改为:2048
4)增加:yarn.nodemanager.aux-services,值改为:mapreduce_shuffle
5)增加:yarn.log-aggregation-enable,值改为:true

2.改mapred-site.xml
配置文件:
#find / -name mapred-site.xml
同样的,同时修改以下三个文件的各参数:
1)/etc/hadoop/conf.cloudera.yarn/mapred-site.xml
2)/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/mapred-site.xml
3)/etc/hive/conf.cloudera.hive/mapred-site.xml
修改参数:
1)修改mapreduce.map.memory.mb,值改为:2048
2)修改mapreduce.reduce.memory.mb,值改为:2048
3)mapreduce.jobhistory.address,值改为:hadoop01:10020
4)mapreduce.jobhistory.webapp.addres,值改为:hadoop01:19888
5)mapreduce.application.classpath,值改为:
hive insert into count(*)报错

3.改/etc/profile
增加HADOOP_HOME,修改PATH增加HADOOP_HOME
修改如下:
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/hadoop
增加export PATH=PATH:PATH:HADOOP_HOME/bin:$HADOOP_HOME/sbin
hive insert into count(*)报错

4.改hive-site.xml
配置文件:
1)/etc/hive/conf.cloudera.hive/hive-site.xml
2)/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/hive/conf.dist/hive-site.xml

修改参数:
1)hive.insert.into.multilevel.dirs,值改为:true

相关文章: