Hadoop2.7实战v1.0之Hive-2.0.0+MySQL远程模式安装-低调大师

Hadoop2.7实战v1.0之Hive-2.0.0+MySQL远程模式安装

2016-04-16 708

环境:Apache Hadoop2.7分布式集群环境(HDFS HA,Yarn HA,HBase HA)

元数据库mysql部署在hadoop-01机器上

user:hive

password:hive

database:hive_remote_meta

hive服务端部署在hadoop-01机器上

hive客户端部署在hadoop-02机器上

1.Install MySQL5.6.23 on hadoop-01

2.Create db and user

hadoop-01:mysqladmin:/usr/local/mysql:>mysql -uroot -p

mysql> create database hive_remote_meta;

Query OK, 1 row affected (0.04 sec)

mysql> create user 'hive' identified by 'hive';

Query OK, 0 rows affected (0.05 sec)

mysql> grant all privileges on hive_remote_meta.* to 'hive'@'%';

Query OK, 0 rows affected (0.03 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.01 sec)

3.在安装Install hive-2.0.0

[root@hadoop-01 tmp]# wget http://apache.communilink.net/hive/hive-2.0.0/apache-hive-2.0.0-bin.tar.gz

[root@hadoop-01 tmp]# tar zxvf apache-hive-2.0.0-bin.tar.gz

[root@hadoop-01 tmp]# mv apache-hive-2.0.0-bin /hadoop/hive-remote-server

[root@hadoop-01 tmp]# cd /hadoop/hive-remote-server

[root@hadoop-01 hive-remote-server]# ll

total 588

drwxr-xr-x 3 root root 4096 Mar 29 23:19 bin

drwxr-xr-x 2 root root 4096 Mar 29 23:19 conf

drwxr-xr-x 4 root root 4096 Mar 29 23:19 examples

drwxr-xr-x 7 root root 4096 Mar 29 23:19 hcatalog

drwxr-xr-x 4 root root 12288 Mar 29 23:19 lib

-rw-r--r-- 1 root root 26335 Jan 22 12:28 LICENSE

-rw-r--r-- 1 root root 513 Jan 22 12:28 NOTICE

-rw-r--r-- 1 root root 4348 Feb 10 09:50 README.txt

-rw-r--r-- 1 root root 527063 Feb 10 09:56 RELEASE_NOTES.txt

drwxr-xr-x 4 root root 4096 Mar 29 23:19 scripts

[root@hadoop-01 hive-remote-server]#

4.Configure profile on hadoop-01

[root@hadoop-01 ~]# vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

export HADOOP_HOME=/hadoop/hadoop-2.7.2

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export HBASE_HOME=/hadoop/hbase-1.2.0

export ZOOKEEPER_HOME=/hadoop/zookeeper

export HIVE_HOME=/hadoop/hive-remote-server

export PATH=.:$HADOOP_HOME/bin:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH

[root@hadoop-01 ~]# source /etc/profile

[root@hadoop-01 ~]#

5.configure jdbc jar

[root@hadoop-01 tmp]# wget http://ftp.nchu.edu.tw/Unix/Database/MySQL/Downloads/Connector-J/mysql-connector-java-5.1.36.tar.gz

[root@hadoop-01 tmp]# tar zxvf mysql-connector-java-5.1.36.tar.gz

[root@hadoop-01 tmp]# cd mysql-connector-java-5.1.36

[root@hadoop-01 mysql-connector-java-5.1.36]# ll

total 1428

-rw-r--r-- 1 root root 90430 Jun 20 2015 build.xml

-rw-r--r-- 1 root root 235082 Jun 20 2015 CHANGES

-rw-r--r-- 1 root root 18122 Jun 20 2015 COPYING

drwxr-xr-x 2 root root 4096 Mar 29 23:35 docs

-rw-r--r-- 1 root root 972009 Jun 20 2015 mysql-connector-java-5.1.36-bin.jar

-rw-r--r-- 1 root root 61423 Jun 20 2015 README

-rw-r--r-- 1 root root 63674 Jun 20 2015 README.txt

drwxr-xr-x 8 root root 4096 Jun 20 2015 src

[root@hadoop-01 mysql-connector-java-5.1.36]# cp mysql-connector-java-5.1.36-bin.jar $HIVE_HOME/lib/

6.Configure Hive服务端

[root@hadoop-01 ~]# cd $HIVE_HOME/conf

[root@hadoop-01 conf]# cp hive-default.xml.template hive-default.xml

# "hdfs://mycluster"是指$HADOOP_HOME/etc/hadoop/core-site.xml文件的fs.defaultFS的值(NameNode HA URI)

#格式:jdbc:mysql:///?createDatabaseIfNotExist=true

点击(此处)折叠或打开

[root@hadoop-01 conf]# vi hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://mycluster/user/hive_remote/warehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://hadoop-01:3306/hive_remote_meta?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>hive.hwi.war.file</name>
<value>${HIVE_HOME}/lib/hive-hwi-2.0.0.jar</value>
<description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </description>
</property>
</configuration>
"hive-site.xml" 26L, 1056C written
[root@hadoop-01 conf]#

7.scp hive to client

[root@hadoop-01 hadoop]# pwd

/hadoop

[root@hadoop-01 hadoop]# scp -r hive-remote-server root@hadoop-02:/hadoop/hive-remote-client

8.Configure profile on hadoop-01

[root@hadoop-01 ~]# vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

export HADOOP_HOME=/hadoop/hadoop-2.7.2

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export HBASE_HOME=/hadoop/hbase-1.2.0

export ZOOKEEPER_HOME=/hadoop/zookeeper

export HIVE_HOME=/hadoop/hive-remote-client

export PATH=.:$HADOOP_HOME/bin:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH

[root@hadoop-01 ~]# source /etc/profile

[root@hadoop-01 ~]#

9.Configure Hive客户端

点击(此处)折叠或打开

[root@hadoop-02 conf]# vi hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://hadoop-01:9083</value>
<description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://mycluster/user/hive_remote/warehouse</value>
</property>
</configuration>
"hive-site.xml" 26L, 1056C written
[root@hadoop-01 conf]#

10.hive服务端如果是第一次需要执行初始化命令：schematool -initSchema -dbType mysql

[root@hadoop-01 bin]# schematool -initSchema -dbType mysql

Metastore connection URL: jdbc:mysql://hadoop-01.telenav.cn:3306/hive_remote_meta?createDatabaseIfNotExist=true

Metastore Connection Driver : com.mysql.jdbc.Driver

Metastore connection User: hive

Starting metastore schema initialization to 2.0.0

Initialization script hive-schema-2.0.0.mysql.sql

Initialization script completed

schemaTool completed

[root@hadoop-01 bin]#

11.启动hive服务端和客户端

【服务端】:

hive --service metastore -p [port]

如果不加端口默认启动：hive --service metastore，则默认监听端口是：9083 ，注意客户端中的端口配置需要和启动监听的端口一致。服务端启动正常后，客户端就可以执行hive操作了

### & 以后台方式启动

[root@hadoop-01 bin]# hive --service metastore &

Starting Hive Metastore Server

【客户端】:

[root@hadoop-02 hive-remote-client]# cd bin

[root@hadoop-02 bin]# hive

Logging initialized using configuration in jar:file:/hadoop/hive-remote-client/lib/hive-common-2.0.0.jar!/hive-log4j2.properties

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.

hive>

12.创建表，load数据验证

## "tab制表符"分隔

[root@hadoop-02 bin]# vi /tmp/studentInfo.txt

1 a 26 110

1 a 26 113

2 b 11 222

[root@hadoop-02 bin]# hive

Logging initialized using configuration in jar:file:/hadoop/hive-remote-client/lib/hive-common-2.0.0.jar!/hive-log4j2.properties

Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.

hive>

> create table studentinfo (id int,name string, age int,tel string)

> row format delimited fields terminated by '\t'

> stored as textfile;

hive> load data local inpath '/tmp/studentInfo.txt' overwrite into table studentinfo;

Loading data to table default.studentinfo

Moved: 'hdfs://mycluster/user/hive_remote/warehouse/studentinfo/studentInfo.txt' to trash at: hdfs://mycluster/user/root/.Trash/Current

Time taken: 2.941 seconds

hive> select * from studentinfo;

1 a 26 113

2 b 11 222

Time taken: 1.607 seconds, Fetched: 2 row(s)

hive

13.查看hdfs文件系统

[root@hadoop-01 bin]# ps -ef|grep hive

root 11629 4509 1 19:39 pts/0 00:00:21 /usr/java/jdk1.7.0_67-cloudera/bin/java -Xmx256m -Djava.library.path=/hadoop/hadoop-2.7.2/lib -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/hadoop/hadoop-2.7.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/hadoop/hadoop-2.7.2 -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dlog4j.configurationFile=hive-log4j2.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /hadoop/hive-remote-server/lib/hive-service-2.0.0.jar org.apache.hadoop.hive.metastore.HiveMetaStore

root 13351 4509 0 19:57 pts/0 00:00:00 grep hive

[root@hadoop-01 bin]# hadoop fs -ls /user/hive_remote/warehouse/studentinfo

Found 1 items

-rwx------ 3 root root 22 2016-04-16 19:54 /user/hive_remote/warehouse/studentinfo/studentInfo.txt

[root@hadoop-01 bin]# hadoop fs -cat /user/hive_remote/warehouse/studentinfo/studentInfo.txt

1 a 26 113

2 b 11 222

[root@hadoop-01 bin]#

14.参考

https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin

微信关注我们

原文链接：https://yq.aliyun.com/articles/578869

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

从 "No module named pyspark" 到远程提交 spark 任务

版权声明：本文为半吊子子全栈工匠（wireless_com，同公众号)原创文章，未经允许不得转载。 https://blog.csdn.net/wireless_com/article/details/51170246 能在本地Mac环境用python提交spark 任务会方便很多，但是在安装了 spark-1.6-bin-without-hadoop (spark.apache.org/download) 之后，在python 中 “import pyspark”会报“no module named pyspark” 错误。没错，这种错误都是路径问题。为了本地使用spark，需要在~/.bash_profile 中增加两个环境变量：SPARK_HOME 以及必知的PYTHONPATH export SPARK_HOME=/Users/abc/Documents/spark-1.6.0-bin-without－hadoop ＃这是spark 的安装路径 export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j...

2016-04-16

999

MapReduce的编程思想(1)

MapReduce的编程思想(1) MapReduce的过程(2) 1. MapReduce采用分而治之的思想，将数据处理拆分为主要的Map（映射）与Reduce（化简）两步，MapReduce操作数据的最小单位是一个键值对。 2. MapReduce计算框架为主从架构，分别是JobTracker与TaskTracker。 JobTracker在集群中为主的角色，它主要负责任务调度和集群资源监控，并不参与具体的计算。 TaskTracker在集群中为从的角色，它主要负责汇报心跳和执行JobTracker的命令（启动任务、提交任务、杀死人物、杀死作业和重新初始化）。客户端，用户编写MapReduce程序通过客户端提交到JobTracker。 3. MapReduce作业（job）是指用户提交的最小单位，而Maop/Reduce任务（task）是MapReduce计算的最小单位。 MapReduce作业由JobTracker的作业分解模块分拆为任务交给各个TaskTracker执行，在MapReduce计算框架中，任务分为两种，Map任务和Reduce任务。 4. MapReduce的计...

2016-04-17

794

发表评论

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。