Upgrade json-serde-xxx jar in Apache Hive-1.2.1
1.背景:
在迁移ETL至ALiYun EMR环境，使用我们定制的Hive-1.2.1-emr版本过程中，业务脚本需要使用json-serde-1.1.9.9-SNAPSHOT-jar-with-dependencies.jar
2.抛错如下:
Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.openx.data.jsonserde.JsonSerDe.
http://stackoverflow.com/questions/26644351/cannot-validate-serde-org-openx-data-jsonserde-jsonserde
3.解决版本:
升级json-serde jar为最新版
1.Configure mvn
[root@hadoopnn-01 hadoop]# export MAVEN_HOME=/hadoop/maven
[root@hadoopnn-01 hadoop]# export PATH=$MAVEN_HOME/bin:$PATH2.Download json-serde source code
[root@hadoopnn-01 hadoop]# mkdir test
[root@hadoopnn-01 hadoop]# cd test
[root@hadoopnn-01 test]# git clone https://github.com/rcongiu/Hive-JSON-Serde.git
3.Compile json-serde
[root@hadoopnn-01 test]# cd Hive-JSON-Serde/
[root@hadoopnn-01 Hive-JSON-Serde]# mvn -Pcdh5 clean package
.................
.................
[INFO] Building jar: /hadoop/test/Hive-JSON-Serde/json-udf/target/json-udf-1.3.8-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] openx-json-serde ................................... SUCCESS [ 1.627 s]
[INFO] json ............................................... SUCCESS [ 4.308 s]
[INFO] json-cdh4-shim ..................................... SUCCESS [ 3.250 s]
[INFO] json-cdh5-shim ..................................... SUCCESS [ 2.800 s]
[INFO] json-serde-main .................................... SUCCESS [ 6.131 s]
[INFO] json-udf ........................................... SUCCESS [ 1.701 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.445 s
[INFO] Finished at: 2016-10-04T11:43:33+08:00
[INFO] Final Memory: 76M/335M
[INFO] ------------------------------------------------------------------------4.sz json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
[root@hadoopnn-01 Hive-JSON-Serde]# cd json-serde/target/
[root@hadoopnn-01 target]# ll
total 180
drwxr-xr-x 2 root root 4096 Oct 4 11:43 archive-tmp
drwxr-xr-x 3 root root 4096 Oct 4 11:43 classes
-rw-r--r-- 1 root root 37935 Oct 4 11:43 json-serde-1.3.8-SNAPSHOT.jar
-rw-r--r-- 1 root root 83787 Oct 4 11:43 json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
-rw-r--r-- 1 root root 29997 Oct 4 11:43 json-serde-1.3.8-SNAPSHOT-tests.jar
drwxr-xr-x 2 root root 4096 Oct 4 11:43 maven-archiver
drwxr-xr-x 2 root root 4096 Oct 4 11:43 surefire
drwxr-xr-x 2 root root 4096 Oct 4 11:43 surefire-reports
drwxr-xr-x 4 root root 4096 Oct 4 11:43 test-classes
[root@hadoopnn-01 target]# sz json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
rz
Starting zmodem transfer. Press Ctrl+C to cancel.
Transferring json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar...
100% 81 KB 81 KB/sec 00:00:01 0 Errors
