Hive JSON Serde MetaStore 问题

Question

我有一个包含 JSON 数据的外部表，我正在使用 JsonSerde 将数据填充到表中。我正确地填充了数据，当我查询数据时，我能够正确地看到结果。

但是，当我在该表上使用 desc 命令时，我得到了所有列 cmets 的 from deserializer 文本。

下面是建表ddl。

  CREATE EXTERNAL TABLE IF NOT EXISTS my_table (
    field1 string COMMENT 'This is a field1', 
    field2 int COMMENT 'This is a field2', 
    field3 string COMMENT 'This is a field3', 
    field4 double COMMENT 'This is a field4'
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde'
    Location '/user/uszszb6/json_test/data';

数据文件中的条目。

{"field1":"data1","field2":100,"field3":"more data1","field4":123.001}
{"field1":"data2","field2":200,"field3":"more data2","field4":123.002}
{"field1":"data3","field2":300,"field3":"more data3","field4":123.003}
{"field1":"data4","field2":400,"field3":"more data4","field4":123.004}

当我使用命令 desc my_table 时，我得到以下输出。

 +-----------+------------+--------------------+--+
    | col_name  | data_type  |      comment       |
    +-----------+------------+--------------------+--+
    | field1    | string     | from deserializer  |
    | field2    | int        | from deserializer  |
    | field3    | string     | from deserializer  |
    | field4    | double     | from deserializer  |
    +-----------+------------+--------------------+--+

JsonSerde 无法正确捕获 cmets。我也尝试过使用其他 JSONSerde，例如

 org.openx.data.jsonserde.JsonSerDe
 org.apache.hive.hcatalog.data.JsonSerDe
 com.amazon.elasticmapreduce.JsonSerde

但是 desc 命令的输出是一样的。这个bug有一张JIRA票[https://issues.apache.org/jira/browse/HIVE-6681][1]

根据票证它已在 0.13 版中解决，我使用的是 hive 1.2.1，但我仍然面临这个问题。

任何人都可以分享您对解决此问题的想法。

Answer 1

是的，看起来这是一个影响所有 Json SerDes 的 Hive 错误，但您是否尝试过使用 DESCRIBE EXTENDED ？

DESCRIBE EXTENDED my_table;

hive> describe extended  json_serde_test;
OK
browser                 string                  from deserializer   
device_uuid             string                  from deserializer   
custom                  struct<customer_id:string>  from deserializer   

Detailed Table Information  
Table(tableName:json_serde_test,dbName:default, owner:rcongiu,
createTime:1448477902, lastAccessTime:0, retention:0, 
sd:StorageDescriptor(cols:[FieldSchema(name:browser, type:string, 
comment:hello), FieldSchema(name:device_uuid, type:string, comment:my 
name is elder price), FieldSchema(name:custom,   
type:struct<customer_id:string>, comment:null)], 
location:hdfs://localhost:9000/user/hive/warehouse/json_serde_test, 
inputFormat:org.apache.hadoop.mapred.TextInputFormat,  
outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, 
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, 
serializationLib:org.openx.data.jsonserde.JsonSerDe, parameters:
{serialization.format=1, mapping.customer_id=Customer ID}), 
bucketCols:[], sortCols:[], parameters:{}, 
skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], 
skewedColValueLocationMaps:{}), storedAsSubDirectories:false), 
partitionKeys:[], parameters:{numFiles=1, 
transient_lastDdlTime=1448477903, COLUMN_STATS_ACCURATE=true, 
totalSize=128, numRows=0, rawDataSize=0}, viewOriginalText:null,     
viewExpandedText:null, tableType:MANAGED_TABLE) 
Time taken: 0.073 seconds, Fetched: 5 row(s)

将输出包含 cmets 的 json-ish 详细描述..有点难以阅读，但它向我展示了 cmets 并且可能足以满足您的目的..或者不是。