Hive - 从时间戳中检索日期时的 to_date vs substr答案

【问题标题】：Hive - to_date vs substr while retrieving date from timestampHive - 从时间戳中检索日期时的 to_date vs substr
【发布时间】：2018-02-11 17:39:48
【问题描述】：

我有一个数据类型为字符串的时间戳列。数据格式为“yyyy-mm-dd hh:mm:ss”。我有两种仅检索日期部分的解决方案。

to_date(col)
substr(col,0,10)

在性能方面，对于庞大的数据量，其中哪一个是更好的解决方案？

【问题讨论】：

标签： hive

【解决方案1】：

我认为您的问题的答案取决于很多事情，但总的来说，查看说明计划是一个不错的起点。在我的测试中，计划似乎没有什么不同。

注意：这是在 Hive 版本 1.1.0-cdh5.12.2 上的 Cloudera 环境中测试的

使用 TO_DATE()：

+----------------------------------------------------+--+
|                      Explain                       |
+----------------------------------------------------+--+
| STAGE DEPENDENCIES:                                |
|   Stage-1 is a root stage                          |
|   Stage-0 depends on stages: Stage-1               |
|                                                    |
| STAGE PLANS:                                       |
|   Stage: Stage-1                                   |
|     Map Reduce                                     |
|       Map Operator Tree:                           |
|           TableScan                                |
|             alias: a                               |
|             Statistics: Num rows: 163043612 Data size: 178714012511 Basic stats: COMPLETE Column stats: NONE |
|             Select Operator                        |
|               expressions: to_date(some_date) (type: string) |
|               outputColumnNames: _col0             |
|               Statistics: Num rows: 163043612 Data size: 178714012511 Basic stats: COMPLETE Column stats: NONE |
|               File Output Operator                 |
|                 compressed: false                  |
|                 Statistics: Num rows: 163043612 Data size: 178714012511 Basic stats: COMPLETE Column stats: NONE |
|                 table:                             |
|                     input format: org.apache.hadoop.mapred.TextInputFormat |
|                     output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat |
|                     serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe |
|                                                    |
|   Stage: Stage-0                                   |
|     Fetch Operator                                 |
|       limit: -1                                    |
|       Processor Tree:                              |
|         ListSink                                   |
|                                                    |
+----------------------------------------------------+--+

使用 SUBSTR()：

+----------------------------------------------------+--+
|                      Explain                       |
+----------------------------------------------------+--+
| STAGE DEPENDENCIES:                                |
|   Stage-1 is a root stage                          |
|   Stage-0 depends on stages: Stage-1               |
|                                                    |
| STAGE PLANS:                                       |
|   Stage: Stage-1                                   |
|     Map Reduce                                     |
|       Map Operator Tree:                           |
|           TableScan                                |
|             alias: b                               |
|             Statistics: Num rows: 163043612 Data size: 178714012511 Basic stats: COMPLETE Column stats: NONE |
|             Select Operator                        |
|               expressions: substr(some_date, 1, 10) (type: string) |
|               outputColumnNames: _col0             |
|               Statistics: Num rows: 163043612 Data size: 178714012511 Basic stats: COMPLETE Column stats: NONE |
|               File Output Operator                 |
|                 compressed: false                  |
|                 Statistics: Num rows: 163043612 Data size: 178714012511 Basic stats: COMPLETE Column stats: NONE |
|                 table:                             |
|                     input format: org.apache.hadoop.mapred.TextInputFormat |
|                     output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat |
|                     serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe |
|                                                    |
|   Stage: Stage-0                                   |
|     Fetch Operator                                 |
|       limit: -1                                    |
|       Processor Tree:                              |
|         ListSink                                   |
|                                                    |
+----------------------------------------------------+--+

【讨论】：