hive2solr时count的一个bug
最近在测试hive导入solr，github上有个相关的代码 https://github.com/chimpler/hive-solr 其原理就是实现inputformat和outputformat，通过mapred来做数据的读写操作。 测试的表结构： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 showcreatetabletable_in_solr1; CREATEEXTERNALTABLEtable_in_solr1( idstringCOMMENT 'fromdeserializer' , cookie_id_sstringCOMMENT 'fromdeserializer' , first_url_sstringCOMMENT 'fromdeserializer' , warehouse_sstringCOMMENT 'fromdeserializer' ) ROWFORMATSERDE 'com.chimpler.hive.solr.SolrSerDe' STOREDBY 'com.chimpler.hive.solr.SolrStorageHandler' WITHSERDEPROPERTIES( 'serialization.format' = '1' , 'solr.column.mapping' = 'id,cookie_id_s,first_url_s,warehouse_s' ) LOCATION 'hdfs://xxxxxx:9000/bip/hive_warehouse/table_in_solr1' TBLPROPERTIES( 'solr.url' = 'http://xxxxxxx:8888/solr/userinfo' , 'transient_lastDdlTime' = '1401357066' , 'solr.buffer.input.rows' = '10000' , 'solr.buffer.output.rows' = '10000' ) 导入数据之后，进行count测试： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 selectcount( 1 )fromtable_in_solr1; EndedJob=job_1401419652664_0010witherrors Errorduringjob,obtainingdebugginginformation... ExaminingtaskID:task_1401419652664_0010_m_000000(andmore)fromjobjob_1401419652664_0010 Taskwiththemostfailures( 4 ): ----- TaskID: task_1401419652664_0010_m_000000 URL: http: //nfzm:8088/taskdetails.jsp?jobid=job_1401419652664_0010&tipid=task_1401419652664_0010_m_000000 ----- DiagnosticMessages for this Task: Error:java.io.IOException:java.lang.NumberFormatException:Forinputstring: "" atorg.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java: 97 ) atorg.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java: 57 ) atorg.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java: 243 ) atorg.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java: 526 ) atorg.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java: 166 ) atorg.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java: 407 ) atorg.apache.hadoop.mapred.MapTask.run(MapTask.java: 340 ) atorg.apache.hadoop.mapred.YarnChild$ 2 .run(YarnChild.java: 160 ) atjava.security.AccessController.doPrivileged(NativeMethod) atjavax.security.auth.Subject.doAs(Subject.java: 396 ) atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java: 1438 ) atorg.apache.hadoop.mapred.YarnChild.main(YarnChild.java: 155 ) Causedby:java.lang.NumberFormatException:Forinputstring: "" atjava.lang.NumberFormatException.forInputString(NumberFormatException.java: 48 ) atjava.lang.Integer.parseInt(Integer.java: 470 ) atjava.lang.Integer.parseInt(Integer.java: 499 ) atcom.chimpler.hive.solr.SolrInputFormat.getReadColumnIDs(SolrInputFormat.java: 38 ) atcom.chimpler.hive.solr.SolrInputFormat.getRecordReader(SolrInputFormat.java: 50 ) atorg.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java: 240 ) ... 9 more FAILED:ExecutionError, return code 2 fromorg.apache.hadoop.hive.ql.exec.MapRedTask MapReduceJobsLaunched: Job 0 :Map: 2 Reduce: 1 HDFSRead: 0 HDFSWrite: 0 FAIL TotalMapReduceCPUTimeSpent: 0 msec 而count(一个字段)是ok的。 从explain的结果来看。 count(字段)对比count(1)增加了Select Operator： 1 2 3 4 5 6 7 8 SelectOperator expressions: expr:id type:string outputColumnNames:id GroupByOperator aggregations: expr:count(id) 从job的出错信息来看，是由于类型转换出错导致，根据堆栈的信息，可以定位到出错的代码片段： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 atcom.chimpler.hive.solr.SolrInputFormat.getReadColumnIDs(SolrInputFormat.java: 38 ) 即： public static final StringREAD_COLUMN_IDS_CONF_STR= "hive.io.file.readcolumn.ids" ; public static final StringREAD_COLUMN_NAMES_CONF_STR= "hive.io.file.readcolumn.names" ; private static final StringREAD_COLUMN_IDS_CONF_STR_DEFAULT= "" ; private static final StringREAD_ALL_COLUMNS= "hive.io.file.read.all.columns" ; private static final boolean READ_ALL_COLUMNS_DEFAULT= true ; /** *Returnsanarrayofcolumnids(startfromzero)whichissetinthegiven *parameter<tt>conf</tt>. */ public static List<Integer>getReadColumnIDs(JobConfconf){ Stringskips=conf.get(READ_COLUMN_IDS_CONF_STR,READ_COLUMN_IDS_CONF_STR_DEFAULT); String[]list=skips.split( "," ); List<Integer>result= new ArrayList<Integer>(list.length); for (Stringelement:list){ //itmaycontainduplicates,removeduplicates IntegertoAdd=Integer.parseInt(element); if (!result.contains(toAdd)){ result.add(toAdd); } } return result; } 即在hive的map阶段，通过SolrInputFormat做数据读取时，hive.io.file.readcolumn.ids的默认值是空，如果没有字段名 ，就会产生空值，在Integer.parseInt(element)就会报错。 这点可以通过下面的代码进行验证: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 import java.io.IOException; import java.util.ArrayList; import java.util.List; public class ttt{ public static void main(String[]args){ Stringskips= "" ; String[]list=skips.split( "," ); System.out.println(list); List<Integer>result= new ArrayList<Integer>(list.length); for (Stringelement:list){ try { IntegertoAdd=Integer.parseInt(element); if (!result.contains(toAdd)){ result.add(toAdd); } } catch (Exceptione){ System.out.println(e); } } } } 本文转自菜菜光 51CTO博客，原文链接：http://blog.51cto.com/caiguangguang/1422733，如需转载请自行联系原作者
