Hive 始终以本地模式运行映射作业答案

【问题标题】：Hive always run mapred jobs in local modeHive 始终以本地模式运行映射作业
【发布时间】：2014-04-29 16:02:43
【问题描述】：

我们正在使用 Hive (0.13.0) 测试多节点 hadoop 集群 (2.4.0)。集群工作正常，但是当我们在 hive 中运行查询时，mapred 作业总是在本地执行。例如：

没有hive-site.xml（其实除了defaults之外没有任何配置文件）我们设置mapred.job.tracker：

hive> SET mapred.job.tracker=192.168.7.183:8032;

然后运行查询：

hive> select count(1) from suricata;

Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
OpenJDK 64-Bit Server VM warning: You have loaded library /hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/04/29 12:48:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
Execution log at: /tmp/hadoopuser/hadoopuser_20140429124747_badfcce6-620e-4718-8c3b-e4ef76bdba7e.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2014-04-29 12:48:05,450 null map = 0%,  reduce = 0%
.......
.......
2014-04-29 12:52:26,982 null map = 100%,  reduce = 100%
Ended Job = job_local1983771849_0001
Execution completed successfully
**MapredLocal task succeeded**
OK
266559841
Time taken: 270.176 seconds, Fetched: 1 row(s)

我们缺少什么？

【问题讨论】：

您是否尝试过更改 core-site.xml 中的 Hadoop 配置？如果你有一个本地 Hadoop 安装并且有一个本地配置 Hive 可能会使用这个..

标签： hive cloudera cloudera-cdh

【解决方案1】：

将 hive.exec.mode.local.auto 设置为 false，这将禁用 Hive 中的本地模式执行

【讨论】：

在hive 控制台中运行SET hive.exec.mode.local.auto=false，得到相同的结果：Job running in-process (local Hadoop)。此设置似乎不会始终禁用本地模式。

【解决方案2】：

对于每个查询，编译器都会生成 map-reduce 作业的 DAG。如果作业在本地模式下运行，请检查以下属性：

mapreduce.framework.name=local;
hive.exec.mode.local.auto=false;

如果启用了自动选项，则 hive 在本地模式下运行作业

Total input size < hive.exec.mode.local.auto.inputbytes.max
Total number of map tasks < hive.exec.mode.local.auto.tasks.max
Total number of reduce tasks =< 1 or 0

这些选项从 0.7 开始提供

【讨论】：