【问题标题】:unix_timestamp is behaving incorrect for Hiveunix_timestamp 对于 Hive 的行为不正确
【发布时间】:2017-10-26 13:51:03
【问题描述】:

我在单个 HQL 中有以下 2 个插入查询,其中我将几个字段从表“Final_table_temp”插入到 Final_table1 和 Final_table2。两个目标表的结构完全相同。

insert into Final_table1
PARTITION(event_date,service_id)
select from_unixtime(unix_timestamp(event_timestamp ,'yyyy-MM-dd HH'), 'yyyy-MM-dd HH:00:00.0'),
from_unixtime(unix_timestamp(event_timestamp ,'yyyy-MM-dd HH'), 'yyyy-MM-dd') as event_date,
service_id 
from Final_table_temp;

insert into Final_table2
PARTITION(event_date,service_id)
select from_unixtime(unix_timestamp(event_timestamp ,'yyyy-MM-dd HH'), 'yyyy-MM-dd HH:00:00.0'),
from_unixtime(unix_timestamp(event_timestamp ,'yyyy-MM-dd HH'), 'yyyy-MM-dd') as event_date,
service_id 
from Final_table_temp;

Final_table_temp 中的 event_timestamp 值:

2017-10-26 22

Final_table1 中的 event_timestamp 值:

2017-10-26 22:00:00.000

Final_table2 中的 event_timestamp 值:

2017-10-26 21:00:00.000

请帮助我理解为什么它会改变 table2 的值。它应该与 table1 相同,因为它们的查询没有变化并且源也相同?


hive> desc extended Final_table1;
OK
event_timestamp         timestamp
event_date              date
service_id              int

# Partition Information
# col_name              data_type               comment

event_date              date
service_id              int

Detailed Table Information      Table(tableName:Final_table1, dbName:rwdb, owner:hdfs, createTime:1496391931, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:event_timestamp, type:timestamp, comment:null), FieldSchema(name:event_date, type:date, comment:null), FieldSchema(name:service_id, type:int, comment:null)], location:hdfs://R333:8020/user/hive/warehouse/rwdb.db/Final_table1, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{field.delim=,, serialization.format=,}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[FieldSchema(name:event_date, type:date, comment:null), FieldSchema(name:service_id, type:int, comment:null)], parameters:{transient_lastDdlTime=1496391931}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)

=======================================================
hive> desc extended Final_table2;
OK
event_timestamp         timestamp
event_date              date
service_id              int

# Partition Information
# col_name              data_type               comment

event_date              date
service_id              int

Detailed Table Information      Table(tableName:Final_table2, dbName:rwdb, owner:hdfs, createTime:1509000492, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:event_timestamp, type:timestamp, comment:null), FieldSchema(name:event_date, type:date, comment:null), FieldSchema(name:service_id, type:int, comment:null)], location:hdfs://R333:8020/user/hive/warehouse/rwdb.db/Final_table2, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{field.delim=,, serialization.format=,}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[FieldSchema(name:event_date, type:date, comment:null), FieldSchema(name:service_id, type:int, comment:null)], parameters:{transient_lastDdlTime=1509000492}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)

【问题讨论】:

  • 你们的桌子在同一个时区吗?
  • 是的,两者都在同一个时区和同一个 Hive 架构中。
  • DESCRIBE EXTENDED TABLEX 的输出是什么?
  • 最后在问题本身中添加了两个表的描述。请检查。

标签: hive unix-timestamp


【解决方案1】:

替换

'yyyy-MM-dd HH:00:00.0' 与 'yyyy-MM-dd HH:mm:ss.S'

【讨论】:

  • 如何改变小时部分?
  • 尝试使用 from_utc_timestamp 而不是 from_unixtime
猜你喜欢
  • 2015-05-07
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2011-08-01
  • 1970-01-01
  • 2011-01-01
相关资源
最近更新 更多