【发布时间】：2015-09-04 12:53:34
【问题描述】：
我将在 HIVE 中加入 3 个巨大的表（十亿行表）。收集了所有统计信息，但性能仍然很差（查询需要 40 分钟）。
我可以在 HIVE 提示中设置任何参数以获得更好的性能吗？
当我尝试执行时，我会看到类似
的信息Sep 4, 2015 7:40:23 AM INFO: parquet.hadoop.ParquetInputFormat: Total input paths to process : 1
Sep 4, 2015 7:40:23 AM INFO: parquet.hadoop.ParquetFileReader: reading another 1 footers
所有表都是在 BigSql 中创建的，存储参数为“STORED AS PARQUETFILE”
如何在 HIVE 查询运行时隐藏作业进度详细信息？
关于HIVE版本
hive> set system:sun.java.command;
system:sun.java.command=org.apache.hadoop.util.RunJar /opt/ibm/biginsights/hive/lib/hive-cli-0.12.0.jar org.apache.hadoop.hive.cli.CliDriver -hiveconf hive.aux.jars.path=file:///opt/ibm/biginsights/hive/lib/hive-hbase-handler-0.12.0.jar,file:///opt/ibm/biginsights/hive/lib/hive-contrib-0.12.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-client-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-common-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-hadoop2-compat-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-prefix-tree-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-protocol-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-server-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/htrace-core-2.01.jar,file:///opt/ibm/biginsights/hive/lib/zookeeper-3.4.5.jar,file:///opt/ibm/biginsights/sheets/libext/piggybank.jar,file:///opt/ibm/biginsights/sheets/libext/pig-0.11.1.jar,file:///opt/ibm/biginsights/sheets/libext/avro-1.7.4.jar,file:///opt/ibm/biginsights/sheets/libext/opencsv-1.8.jar,file:///opt/ibm/biginsights/sheets/libext/json-simple-1.1.jar,file:///opt/ibm/biginsights/sheets/libext/joda-time-1.6.jar,file:///opt/ibm/biginsights/sheets/libext/bigsheets.jar,file:///opt/ibm/biginsights/sheets/libext/bigsheets-serdes-1.0.0.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-column-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-common-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-encoding-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-generator-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-hadoop-bundle-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-hive-bundle-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-thrift-1.3.2.jar,file:///opt/ibm/biginsights/hive/lib/guava-11.0.2.jar【问题讨论】：
-
如果你在所有 3 个表中加入 100% 的记录，那么统计数据不会有太大的不同。 40 分钟的大规模 Hive 查询并不令人惊讶。一些兴趣点可能是：MapReduce 或 Tez？什么样的加入（参见EXPLAIN）？每个连接步骤有多少个映射器（参见 MR 或 Tez 日志）？大量数据倾斜（参见一些映射器比其他映射器咀嚼更多数据并减慢步骤）？压缩中间结果？等等等等等等等等等等等等。
-
当我使用 STORED PARAMETER 作为 ORC 而不是 PARQUET 时，我得到了非常好的性能提升。你能告诉我为什么会这样吗？
-
为什么选择 Hive+Parquet？因为 Cloudera 营销炒作？请参阅我对其他帖子（和帖子顺便说一句）stackoverflow.com/questions/32350664/… 的评论
标签： performance hive