Author: kwu 

【解决】hive动态添加partitions不能超过100的问题,全量动态生成partitions超过100会出现例如以下异常:

The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

解决100限制,可设置例如以下參数:

set hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions=100000; 
SET hive.exec.max.dynamic.partitions.pernode=100000; 
insert overwrite table ods.cms_entity PARTITION (DAY)
select 
ENTITY_ID  ,
ENTITY_NAME  ,
ENTITY_DESC  ,
ENTITY_TYPE  ,
ENTITY_PID  ,
ENTITY_TIME  ,
ENTITY_PRIORITY  ,
ENTITY_STATUS  ,
ENTITY_CHANNEL  ,
ENTITY_EDITOR  ,
ENTITY_TEMPLATE  ,
ENTITY_URL    ,
ENTITY_CATEGORY  ,
ENTITY_PARAM  ,
ENTITY_SHORTNAME  ,
ENTITY_SUBTYPE  ,
ENTITY_COMPDELAY  ,
day
from stage.cms_entity_by_day;


设置动态partitions为100000,该语句插入几年按天的partitions超过1000个,成功运行语句。

相关文章:

  • 2021-12-03
  • 2021-08-21
  • 2021-07-08
  • 2022-12-23
  • 2022-12-23
  • 2022-01-05
  • 2021-09-25
猜你喜欢
  • 2021-09-07
  • 2022-12-23
  • 2021-05-19
  • 2021-07-20
  • 2022-12-23
  • 2021-12-09
  • 2022-12-23
相关资源
相似解决方案