热门搜索:
您现在的位置是：首页 > 文章详情

【原创】hive关联hbase表后导致统计数据报错

日期：2012-06-19点击：792
环境说明：
搭建好的hadoop+hbase+zookeeper集群，因为hbase里面查询数据不支持select语句，所以搭建起了hive（数据仓库）。我的hive搭建过程也不做太多的介绍，用的是第三方数据库mysql存储hive的元数据。在hive里面我把hbase数据库的xyz表和hive里面的hbase_table_1表关联上，然后执行select * from table可以查到数据，但是select count(*) from table死活报错，结果是mapreduce的任务没跑成功。截图如下：
先查看hbase数据库的xyz表的数据
hbase(main):001:0> scan 'xyz'
ROW                  COLUMN+CELL                                               
10000                column=cf1:val, timestamp=1340091488116, value=China     
1 row(s) in 0.6730 seconds
hbase(main):002:0>
其次查看hive中的hbase_table_1表的数据
hive> select * from hbase_table_1;
OK
10000   China
Time taken: 4.133 seconds
hive>
最后我在hive里要做统计多少行命令和报错信息
hive> select count(*) from  hbase_table_1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapred.reduce.tasks=
Starting Job = job_201206190956_0001, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201206190956_0001
Kill Command = /opt/hadoop/libexec/../bin/hadoop job  -Dmapred.job.tracker=master:9002 -kill job_201206190956_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-06-20 10:46:58,214 Stage-1 map = 0%,  reduce = 0%
2012-06-20 10:47:58,795 Stage-1 map = 0%,  reduce = 0%
2012-06-20 10:48:03,875 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201206190956_0001 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201206190956_0001_m_000002 (and more) from job job_201206190956_0001
Exception in thread "Thread-36" java.lang.RuntimeException: Error while reading from task log url
        at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
        at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:211)
        at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://slave1:50060/tasklog?taskid=attempt_201206190956_0001_m_000000_1&start=-8193
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1305)
        at java.net.URL.openStream(URL.java:1009)
        at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
        ... 3 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
经过几天上网查询和同事沟通的结果，解决上述问题需要有两个步骤：
1、在hive的配置文件hive-site.xml里面增加如下内容，当然value里面的值根据你自己的实际情况来写
 
    hive.aux.jars.path
 file:///opt/hive/lib/hive-hbase-handler-0.8.1.jar,file:///opt/hive/lib/h
base-0.92.1.jar,file:///opt/hive/lib/zookeeper-3.3.1.jar
 
2、然后将namenode节点的hbase配置文件hbase-site.xml拷贝到hadoop的conf目录下，最后将你的它用rsync同步到所有的datanode节点上。
最后我们在查一下试试？
hive> select count(*) from hbase_table_1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapred.reduce.tasks=
Starting Job = job_201206190956_0003, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201206190956_0003
Kill Command = /opt/hadoop/libexec/../bin/hadoop job  -Dmapred.job.tracker=master:9002 -kill job_201206190956_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-06-20 12:04:12,499 Stage-1 map = 0%,  reduce = 0%
2012-06-20 12:04:27,668 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:28,682 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:29,703 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:30,713 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:31,724 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:32,734 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:33,757 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:34,768 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:35,777 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:36,788 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:37,798 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:38,808 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:39,869 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:40,880 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:42,126 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:43,136 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:44,145 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:45,155 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:46,164 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:47,174 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:48,183 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:49,236 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:50,247 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:51,267 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:52,277 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:53,288 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:54,320 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:55,330 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:56,341 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:57,364 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:58,375 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
MapReduce Total cumulative CPU time: 7 seconds 480 msec
Ended Job = job_201206190956_0003
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   Accumulative CPU: 7.48 sec   HDFS Read: 240 HDFS Write: 2 SUCESS
Total MapReduce CPU Time Spent: 7 seconds 480 msec
OK
1
Time taken: 92.04 seconds
可以了！因为我的表中只有一行数据！用的虚拟机比较慢，哎！！！
 


 
原文链接：https://yq.aliyun.com/articles/578656
关注公众号

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。

持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

转载内容版权归作者及来源网站所有，本站原创内容转载请注明来源。

相关文章

    文章评论

    共有0条评论来说两句吧...

    文章二维码

    扫描即可查看该文章

    点击排行

    推荐阅读

    最新文章