hive0.13.1 orc兼容性问题fix
线上使用cdh4.6.0和hive0.13.1,在hive0.11之后开始支持orcfile,hive0.13.1使用的是pb2.5.0,cdh4.6.0是用的2.4.0a，线上测试orcfile，建表正常，但是插入数据时报错报错： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 java.lang.VerifyError: class org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndexoverrides final methodgetUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; atjava.lang.ClassLoader.defineClass1(NativeMethod) atjava.lang.ClassLoader.defineClass(UnknownSource) atjava.security.SecureClassLoader.defineClass(UnknownSource) atjava.net.URLClassLoader.defineClass(UnknownSource) atjava.net.URLClassLoader.access$ 100 (UnknownSource) atjava.net.URLClassLoader$ 1 .run(UnknownSource) atjava.net.URLClassLoader$ 1 .run(UnknownSource) atjava.security.AccessController.doPrivileged(NativeMethod) atjava.net.URLClassLoader.findClass(UnknownSource) atjava.lang.ClassLoader.loadClass(UnknownSource) atsun.misc.Launcher$AppClassLoader.loadClass(UnknownSource) atjava.lang.ClassLoader.loadClass(UnknownSource) atorg.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java: 129 ) atorg.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java: 369 ) atorg.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java: 103 ) atorg.apache.hadoop.hive.ql.exec.Utilities.createEmptyFile(Utilities.java: 3065 ) atorg.apache.hadoop.hive.ql.exec.Utilities.createDummyFileForEmptyPartition(Utilities.java: 3089 ) atorg.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java: 3013 ) atorg.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java: 369 ) atorg.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java: 136 ) atorg.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java: 153 ) atorg.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java: 85 ) atorg.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java: 72 ) 从错误可以看出是由于pb兼容问题导致，尝试使用pb2.4.0a编译hive0.13.1: 1.pom.xml文件： 更改 1 <protobuf.version> 2.5 . 0 </protobuf.version> 为 1 <protobuf.version> 2.4 .0a</protobuf.version> 2.运行mvn进行编译： 1 mvnclean package -DskipTests-Phadoop- 2 -Pdist 发现报如下错误： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [ERROR]/home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/gen-java/ org/apache/hadoop/hive/ql/io/orc/OrcProto.java:[ 9828 , 38 ]cannotfindsymbolsymbol : class Parserlocation: package com.google.protobuf[ERROR] /home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/gen-java/org/apache/hadoop /hive/ql/io/orc/OrcProto.java:[ 9839 , 31 ]cannotfindsymbolsymbol: class Parserlocation: package com.google.protobuf .... [ERROR]/home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/gen-java/ org/apache/hadoop/hive/ql/io/orc/OrcProto.java:[ 169 , 9 ]getUnknownFields() inorg.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatisticscannotoverride getUnknownFields()incom.google.protobuf.GeneratedMessage;overriddenmethod is final [ERROR]/home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/ gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java:[ 189 , 20 ] cannotfindsymbolsymbol:methodparseUnknownField (com.google.protobuf.CodedInputStream,com.google.protobuf.UnknownFieldSet. Builder,com.google.protobuf.ExtensionRegistryLite, int ) 3.这里 1 hive_0.13_debug/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java 是由protoc2.5.0生成的，但是jar包却是2.4.0a的，因此会报错 4.考虑使用2.4.0a的protoc手动生成java代码 1 2 protoc--version libprotoc 2.4 .0a 这里主要涉及两个proto文件： 1 2 3 protocql /src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto .proto--java_out=ql /src/gen/protobuf/gen-java/ protoc. /hcatalog/storage-handlers/hbase/src/protobuf/org/apache/hcatalog/hbase/snapshot/RevisionManagerEndpoint .proto --java_out=hcatalog /storage-handlers/hbase/src/gen-java/ 5.重新编译成功，将hive-exec jar包解压后，可以看到pom.properties中pb的版本已经是2.4.0a了： 1 2 3 4 5 6 META-INF /maven/com .google.protobuf /protobuf-java/pom .properties #GeneratedbyMaven #ThuMar1011:50:31CST2011 version=2.4.0a groupId=com.google.protobuf artifactId=protobuf-java 更换hive-exec jar包之后，测试向orc file的表插入数据也正常。 本文转自菜菜光 51CTO博客，原文链接：http://blog.51cto.com/caiguangguang/1599128，如需转载请自行联系原作者
