hive 动态分区(Dynamic Partition)异常处理
Changing Hive Dynamic Partition Limits Symptoms: Hive enforces limits on the number of dynamic partitions that it creates. The default is 100 dynamic partitions per node, with a total (default) limit of 1000 dynamic partitions across all nodes. However, this can be tuned. If your job tries to create too many dynamic partitions, you may see a stack trace similar to the following: Diagnostic Messages for this Task:Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:415)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:747)at org.apache.hadoop.hive.ql.exec.FileSinkOperator.startGroup(FileSinkOperator.java:829)at org.apache.hadoop.hive.ql.exec.Operator.defaultStartGroup(Operator.java:498)at org.apache.hadoop.hive.ql.exec.Operator.startGroup(Operator.java:521)at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:232)... 7 moreFAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTaskMapReduce Jobs Launched: Cause: Your Hive job tried to create too many dynamic partitions on a node, exceeding the current limits. The number of dynamic partitions is controlled by the settings forhive.exec.max.dynamic.partitionsandhive.exec.max.dynamic.partitions.pernode. The default is 100 dynamic partitions on a single node, with a limit of 1000 overall. Solution: Set the values forhive.exec.max.dynamic.partitionsandhive.exec.max.dynamic.partitions.pernodeto higher values. For example: SET hive.exec.dynamic.partition=true;SET hive.exec.max.dynamic.partitions=2048;SET hive.exec.max.dynamic.partitions.pernode=256;
