Hive报错, [Fatal Error] total number of created files now is 100028, which exceeds 100000.
问题1:
Hive Dynamic partition Error : [Fatal Error] total number of created files now is 100028, which exceeds 100000. Killing the job.
Solution:
So my config increases the default partitions and files created limit:
set hive.exec.dynamic.partition=true;
set hive.exec.max.dynamic.partitions=100000;
SET hive.exec.max.dynamic.partitions.pernode=100000;
set hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.created.files=900000;