0010-Hive多分隔符支持示例-低调大师

0010-Hive多分隔符支持示例

2018-11-16 627

温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。

1.问题描述

如何将多个字符作为字段分割符的数据文件加载到Hive表中，事例数据如下：

字段分隔符为“@#$”

test1@#$test1name@#$test2value
test2@#$test2name@#$test2value
test3@#$test3name@#$test4value

如何将上述事例数据加载到Hive表(multi_delimiter_test)中，表结构如下：

字段名	字段类型
s1	String
s2	String
s3	String

2.Hive多分隔符支持

Hive在0.14及以后版本支持字段的多分隔符，参考https://cwiki.apache.org/confluence/display/Hive/MultiDelimitSerDe

3.实现方式

测试环境说明

测试环境为CDH5.11.1Hive版本为1.1.0操作系统为RedHat6.5

操作步骤

1.准备多分隔符文件并装载到HDFS对应目录

[ec2-user@ip-172-31-8-141  ~]$ cat multi_delimiter_test.dat

test1@#$test1name@#$test2value

test2@#$test2name@#$test2value

test3@#$test3name@#$test4value  

[ec2-user@ip-172-31-8-141  ~]$ hadoop dfs -put multi_delimiter_test.dat /fayson/multi_delimiter_test

[ec2-user@ip-172-31-8-141  ~]$ hadoop dfs -ls /fayson/multi_delimiter_test

DEPRECATED: Use of this  script to execute hdfs command is deprecated.

Instead use the hdfs  command for it.


Found 1 items

-rw-r--r--   3 user_r supergroup         93 2017-08-23 03:24  /fayson/multi_delimiter_test/multi_delimiter_test.dat

[ec2-user@ip-172-31-8-141  ~]$

2.基于准备好的多分隔符文件建表

create  external table multi_delimiter_test(

s1 string,

s2 string,

s3 string)

ROW FORMAT  SERDE 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH  SERDEPROPERTIES ("field.delim"="@#$")

stored as  textfile location '/fayson/multi_delimiter_test';

3.测试

2: jdbc:hive2://localhost:10000/default>  select * from multi_delimiter_test;
+--------------------------+--------------------------+--------------------------+--+
|  multi_delimiter_test.s1  |  multi_delimiter_test.s2  |  multi_delimiter_test.s3  |
+--------------------------+--------------------------+--------------------------+--+
| test1                    | test1name                | test2value               |
| test2                    | test2name                | test2value               |
| test3                    | test3name                | test4value               |
+--------------------------+--------------------------+--------------------------+--+

2:  jdbc:hive2://localhost:10000/default> select count(*) from  multi_delimiter_test;
INFO  : Ended Job = job_1503469952834_0006
INFO  : MapReduce Jobs Launched:
INFO  : Stage-Stage-1: Map: 1  Reduce: 1    Cumulative CPU: 3.25 sec   HDFS  Read: 6755 HDFS Write: 2 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 3 seconds  250 msec
INFO  : Completed executing command(queryId=hive_20170823041818_ce58aae2-e6db-4eed-b6af-652235a6e66a);  Time taken: 33.286 seconds
INFO  : OK
+------+--+
| _c0  |
+------+--+
| 3    |
+------+--+
1 row selected (33.679  seconds)
2:  jdbc:hive2://localhost:10000/def

4.常见问题

1.执行count查询时报错

异常日志

通过beeline执行count查询时报错

2:  jdbc:hive2://localhost:10000/default> select count(*) from  multi_delimiter_test;

INFO  : Compiling  command(queryId=hive_20170823035959_f1b11a9b-757d-4d9b-b8a7-6d4ab1c00a97):  select count(*) from multi_delimiter_test

INFO  : Semantic Analysis Completed

INFO  : Returning Hive schema:  Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)],  properties:null)

INFO  : Completed compiling  command(queryId=hive_20170823035959_f1b11a9b-757d-4d9b-b8a7-6d4ab1c00a97);  Time taken: 0.291 seconds

INFO  : Executing  command(queryId=hive_20170823035959_f1b11a9b-757d-4d9b-b8a7-6d4ab1c00a97):  select count(*) from multi_delimiter_test

INFO  : Query ID =  hive_20170823035959_f1b11a9b-757d-4d9b-b8a7-6d4ab1c00a97

INFO  : Total jobs = 1

INFO  : Launching Job 1 out of 1

INFO  : Starting task [Stage-1:MAPRED] in serial  mode

INFO  : Number of reduce tasks determined at  compile time: 1

INFO  : In order to change the average load for a  reducer (in bytes):

INFO  :    set hive.exec.reducers.bytes.per.reducer=<number>

INFO  : In order to limit the maximum number of  reducers:

INFO  :    set hive.exec.reducers.max=<number>

INFO  : In order to set a constant number of  reducers:

INFO  :    set mapreduce.job.reduces=<number>

INFO  : number of splits:1

INFO  : Submitting tokens for job:  job_1503469952834_0002

INFO  : Kind: HDFS_DELEGATION_TOKEN, Service:  ha-hdfs:nameservice1, Ident: (token for hive: HDFS_DELEGATION_TOKEN  owner=hive/ip-172-31-8-141.ap-southeast-1.compute.internal@CLOUDERA.COM,  renewer=yarn, realUser=, issueDate=1503475160778, maxDate=1504079960778,  sequenceNumber=27, masterKeyId=9)

INFO  : The url to track the job:  http://ip-172-31-9-186.ap-southeast-1.compute.internal:8088/proxy/application_1503469952834_0002/

INFO  : Starting Job = job_1503469952834_0002,  Tracking URL =  http://ip-172-31-9-186.ap-southeast-1.compute.internal:8088/proxy/application_1503469952834_0002/

INFO  : Kill Command =  /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/hadoop/bin/hadoop  job  -kill job_1503469952834_0002

INFO  : Hadoop job information for Stage-1:  number of mappers: 1; number of reducers: 1

INFO  : 2017-08-23 03:59:32,039 Stage-1 map =  0%,  reduce = 0%

INFO  : 2017-08-23 04:00:08,106 Stage-1 map =  100%,  reduce = 100%

ERROR : Ended Job =  job_1503469952834_0002 with errors

ERROR : FAILED: Execution  Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

INFO  : MapReduce Jobs Launched:

INFO  : Stage-Stage-1: Map: 1  Reduce: 1    HDFS Read: 0 HDFS Write: 0 FAIL

INFO  : Total MapReduce CPU Time Spent: 0 msec

INFO  : Completed executing  command(queryId=hive_20170823035959_f1b11a9b-757d-4d9b-b8a7-6d4ab1c00a97);  Time taken: 48.737 seconds

Error: Error while  processing statement: FAILED: Execution Error, return code 2 from  org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)

使用Hive的shell操作报错如下

Error:  java.lang.RuntimeException: Error in configuring object
        at  org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
        at  org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at  org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at  org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at  java.security.AccessController.doPrivileged(Native Method)
        at  javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
        at  org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by:  java.lang.reflect.InvocationTargetException
        at  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at  java.lang.reflect.Method.invoke(Method.java:606)
        at  org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
        ... 9 more
Caused by:  java.lang.RuntimeException: Error in configuring object
        at  org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
        at  org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
        at  org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at  org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
        ... 14 more
Caused by:  java.lang.reflect.InvocationTargetException
        at  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at  java.lang.reflect.Method.invoke(Method.java:606)
        at  org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
        ... 17 more
Caused by:  java.lang.RuntimeException: Map operator initialization failed
        at  org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147)
        ... 22 more
Caused by:  org.apache.hadoop.hive.ql.metadata.HiveException:  java.lang.ClassNotFoundException: Class  org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe not found
        at  org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:323)
        at  org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:333)
        at  org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
        ... 22 more
Caused by:  java.lang.ClassNotFoundException: Class  org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe not found
        at  org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
        at  org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:140)
        at  org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:297)
        ... 24 more
FAILED: Execution Error,  return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map:  1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time  Spent: 0 ms

问题原因分析

org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe类是hive-contrib.jar包里。

在执行非聚合类操作查询时，sql能正常执行，在进行聚合类函数操作时报错，说明在执行MapReduce任务时缺少jar依赖包；MapReduce属于yarn作业，所以yarn运行环境缺少hive-contrib.jar的依赖包。

解决方法

在CDH集群的所有节点一下操作，将hive-contrib-1.1.0-cdh5.10.2.jar包拷贝到yarn的lib目录下

sudo scp -r  /opt/cloudera/parcels/CDH/lib/hive/lib/hive-contrib-1.1.0-cdh5.10.2.jar  /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/

重新运行count语句，执行成功

醉酒鞭名马，少年多浮夸！岭南浣溪沙，呕吐酒肆下！挚友不肯放，数据玩的花！
温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。

微信关注我们

原文链接：https://yq.aliyun.com/articles/670628

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

Apache Flink 漫谈系列(08) - SQL概览

SQL简述 SQL是Structured Query Language的缩写，最初是由美国计算机科学家Donald D. Chamberlin和Raymond F. Boyce在20世纪70年代早期从 Early History of SQL 中了解关系模型后在IBM开发的。该版本最初称为[SEQUEL: A Structured English Query Language]（结构化英语查询语言），旨在操纵和检索存储在IBM原始准关系数据库管理系统System R中的数据。SEQUEL后来改为SQL，因为“SEQUEL”是英国Hawker Siddeley飞机公司的商标。我们看看这款用于特技飞行的英国皇家空军豪客Siddeley Hawk T.1A (Looks great): 第一款SQL数据库在20世纪70年代后期，Oracle

2018-11-17

717

温馨提示：要看高清无码套图，请使用手机打开并单击图片放大查看。 1.文档编写目的本文档讲述如何开发Hive自定义函数（UDF），以及如何在Impala中使用Hive的自定义函数，通过本文档，您将学习到以下知识： 1.如何使用Java开发Hive的自定义函数 2.如何在Hive中创建自定义函数及使用 3.如何在Impala中使用Hive的自定义函数这篇文档将重点介绍UDF在Hive和Impala的使用，并基于以下假设： 1.集群环境正常运行 2.集群安装Hive和Impala服务以下是本次测试环境，但不是本操作手册的硬限制： 1.操作系统：Redhat6.5 2.CDH和CM版本为5.11.1 3.采用sudo权限的ec2-user用户进行操作 2.UDF函数开发使用Intellij工具开发Hive的UDF函数，进行编译； 1.使用Intellij工具通过Maven创建一个Java工程 2.pom.xml文件中增加Hive包的依赖 <dependency> <groupId>org.apache.hive</groupId> <artif...

2018-11-17

562

资源下载

更多资源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。