您现在的位置是：首页 > 文章详情

hadoop-spark-hive-hbase配置相关说明

日期：2015-10-09点击：531收藏

1. zookeeper

配置

cp app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo_sample.cfg app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg
vim app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg
dataDir=/home/cdh5/tmp/zookeeper
clientPort=2183
server.1=ocdata09:2888:3888
mkdir -p /home/cdh5/tmp/zookeeper
vim /home/cdh5/tmp/zookeeper/myid
echo "1" > /home/cdh5/tmp/zookeeper/myid

初始化操作：

或者 ./runRemoteCmd.sh '~/och200/zookeeper/bin/zkServer-initialize.sh --myid=1' zoo

分发配置

./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/ zoo
./runRemoteCmd.sh "app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/zkServer.sh start" zoo
./runRemoteCmd.sh 'echo ruok | nc localhost 2183' zoo
./runRemoteCmd.sh "app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/zkServer.sh stop" zoo

启动

验证

停止

2. HDFS

配置hadoop
hdfs-site.xml

<property>
<name>dfs.nameservices</name>
<value>cdh5cluster</value>
<description>
Comma-separated list of nameservices.
</description>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:50011</value>
<description>
The datanode server address and port for data transfer.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:50076</value>
<description>
The datanode http server address and port.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.datanode.ipc.address</name>
<value>0.0.0.0:50021</value>
<description>
The datanode ipc server address and port.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
（）
<name>dfs.nameservices</name>
<value>cdh5cluster</value>
</property>
<property>
（命名空间中所有NameNode的唯一标示名称。可以配置多个，使用逗号分隔。该名称是可以让DataNode知道每个集群的所有NameNode.当前，每个集群最多只能配置两个NameNode）
<name>dfs.ha.namenodes.cdh5cluster</name>
<value>nn1,nn2</value>
<description></description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data1/cdh5/dfs/name</value>
<description>Determines where on the local filesystem the DFS name node should store the name table.If this is a comma-delimited list of directories,then name table is replicated in all of the directories,for redundancy.</description>
<final>true</final>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data1/cdh5/dfs/data,file:///data2/cdh5/dfs/data,file:///data3/cdh5/dfs/data</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permission</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
<description>
Boolean which enables backend datanode-side support for the experimental DistributedFileSystem*getFileVBlockStorageLocations API.
</description>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
<description>
If "true", enable permission checking in HDFS.
If "false", permission checking is turned off,
but all other behavior is unchanged.
Switching from one parameter value to the other does not change the mode,
owner or group of files or directories.
</description>
</property>
<property>
(每个NAMENODE监听的RPC地址)
<name>dfs.namenode.rpc-address.cdh5cluster.nn1</name>
<value>ocdata09:8030</value>
<description>节点NN1的RPC地址</description>
</property>
<property>
<name>dfs.namenode.rpc-address.cdh5cluster.nn2</name>
<value>ocdata08:8030</value>
<description>节点NN2的RPC地址</description>
</property>
<property>
<name>dfs.namenode.http-address.cdh5cluster.nn1</name>
<value>ocdata09:50082</value>
<description>节点NN1的HTTP地址</description>
</property>
<property>
<name>dfs.namenode.http-address.cdh5cluster.nn2</name>
<value>ocdata08:50082</value>
<description>节点NN2的HTTP地址</description>
</property>
<property>
（这是NameNode读写JNs的uri。通过这个uri，NameNodes可以读写edit log内容。URI的格式”qjournal://host1:port1;host2:port2;host3:port3/journalId”。这里的host1、host2、host3指的是Journal Node的地址，这里必须是奇数个，至少3个;其中journaId是集群的唯一标示符，对于多个联邦命名空间，也使用同一个journaId。配置如下J
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://ocdata05:8488;ocdata06:8488;ocdata07:8488/cdh5cluster</value>
<description>采用3个journalnode节点存储元数据，这是IP与端口</description>
</property>

<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/cdh5/journaldata/jn</value>
<description>journaldata的存储路径</description>
</property>
<property>
<name>dfs.journalnode.rpc-address</name>
<value>0.0.0.0:8488</value>
</property>
<property>
<name>dfs.journalnode.http-address</name>
<value>0.0.0.0:8483</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.cdh5cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
<description>该类用来判断哪个namenode处于生效状态</description>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>shell(/bin/true)</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>10000</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
<description>
Whether automatic failover is enabled. See the HDFS High
Availability documentation for details on automatic HA
configuration.
</description>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>ocdata09:2183</value>
<description>1个zookeeper节点</description>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>4096</value>
<description>
Specifies the maximum number of threads to use for transferring data
in and out of the DN.
</description>
</property>
<property>
<name>dfs.blocksize</name>
<value>64m</value>
<description>
The default block size for new files, in bytes.
You can use the following suffix (case insensitive):
k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),
Or provide complete size in bytes (such as 134217728 for 128 MB).
</description>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>20</value>
<description>The number of server threads for the namenode.</description>
</property>
<property>
（这是一个描述集群中NameNode节点的URI(包括协议、主机名称\端口号—)集群里面的每一台机器都要知道NAMENODE的地址。DataNode节点会先在NAMENODE上注册，这样它们的数据才可以被使用。独立的客户端程序通过这个URI跟DATANODE交互，以取得文件的块列表。）
<name>fs.defaultFS</name>
<value>hdfs://cdh5cluster</value>
</property>
<property>
（hadoop文件系统依赖的基础配置，很多路径都依赖它。如果hdfs-site.xml中不配置namenode和datanode的存放位置，默认就放在这个路径中）
<name>hadoop.tmp.dir</name>
<value>/home/cdh5/tmp/hadoop/hadoop-${user.name}</value>
</property>
<property>

core-site.xml

(对本地jar包进行加载)

<name>io.native.lib.available</name>
<value>true</value>
<description>Should native hadoop libraries, if present, be used.</description>
</property>
(压缩和解压编码类列表，用逗号分隔，这些类是使用java ServiceLoader加载，如果不设置就为null)
<property>
<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
ocdata05
ocdata06
ocdata07
ocdata08
ocdata09
ocdata05
ocdata06
export JAVA_HOME=/home/cdh5/app/jdk1.7.0_21
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc all

slaves

masters

hadoop-env.sh

分发

初始化HDFS：

主节点执行

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs zkfc -formatZK

./runRemoteCmd.sh 'app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh start journalnode' jn

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs namenode -format -initializeSharedEdits

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh start namenode

备节点执行

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs namenode -bootstrapStandby

完成

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh stop namenode

./runRemoteCmd.sh 'app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh stop journalnode' jn

启动HDFS

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/start-dfs.sh
http://10.1.253.99:50082/dfshealth.html (active)
http://10.1.253.98:50082/dfshealth.html (standby)
http://10.1.253.97:8483/journalstatus.jsp
http://10.1.253.96:8483/journalstatus.jsp
http://10.1.253.95:8483/journalstatus.jsp
app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/stop-dfs.sh

验证：

停止HDFS

3. Yarn

配置YARN

mapred-site.xml

cp app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml.template app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml
vim app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>8350</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>0.0.0.0:10121</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>0.0.0.0:19868</value>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>0.0.0.0:50330</value>
</property>
<property>
<name>mapreduce.tasktracker.http.address</name>
<value>0.0.0.0:50360</value>
</property>
vim app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/yarn-site.xml
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-rm-cluster</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<description>Id of the current ResourceManager. Must be set explicitly on each ResourceManager to the appropriate value.</description>
<name>yarn.resourcemanager.ha.id</name>
<value>rm1</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>ocdata09:2183</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>ocdata09:2183</value>
</property>
<property>
<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
<value>5000</value>
</property>
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>ocdata08:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>ocdata08:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>ocdata08:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>ocdata08:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>ocdata08:23141</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm1</name>
<value>ocdata08:23142</value>
</property>
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>ocdata09:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>ocdata09:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>ocdata09:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>ocdata09:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>ocdata09:23141</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm2</name>
<value>ocdata09:23142</value>
</property>
<property>
<description>Address where the localizer IPC is.</description>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:23344</value>
</property>
<property>
<description>NM Webapp address.</description>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:23999</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/tmp/pseudo-dist/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/tmp/pseudo-dist/yarn/log</value>
</property>
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc all
app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/start-yarn.sh

yarn-site.xml

分发

Yarn的启动停止 YARN不需要初始化，登录主节点执行

cdh5 yarn的ha需要手动启动备节点

./runRemoteCmd.sh "cd app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin; ./yarn-daemon.sh start resourcemanager" rm2

验证

http://10.1.253.98:23188/cluster (有节点列表，active)

http://10.1.253.99:23188/cluster (无节点列表，standby)

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hadoop jar app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar TestDFSIO -write -nrFiles 40 -fileSize 20MB

停止YARN

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/stop-yarn.sh

手动停止备节点

./runRemoteCmd.sh "cd app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin; ./yarn-daemon.sh stop resourcemanager" rm2

4. hive

配置

cp hive-env.sh.template hive-env.sh
vim hive-env.sh
export HADOOP_HOME=/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT
cp hive-default.xml.template hive-site.xml
vim hive-site.xml

删除其他配置项，只保留：

（配置元数据库,一般为mysql）

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://10.1.252.69:3306/cdh5?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=UTF-8</value>

</property>

(配置元数据库的Driver)

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

(配置元数据库的名称)

<name>javax.jdo.option.ConnectionUserName</name>

<description>username to use against metastore database</description>

</property>

(配置元数据库的密码)

<name>javax.jdo.option.ConnectionPassword</name>

<description>password to use against metastore database</description>

</property>

元数据库配置

CREATE USER cdh5 IDENTIFIED BY 'cdh5';
CREATE DATABASE cdh5;
alter database cdh5 character set latin1;
grant all privileges on *.* to cdh5@"%" identified by "cdh5";
flush privileges;
scp mysql-connector-java-5.1.26.jar cdh5@10.1.253.99:/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hive-0.12.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/lib/
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hive-0.12.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT app/ochadoop-och3.0.0-SNAPSHOT/ hive
nohup ./hiveserver2 &
jdbc:
jdbc:hive2://10.1.253.99:10000/default
org.apache.hive.jdbc.HiveDriver
lib: Hadoop和hive下所有jar包
!connect jdbc:hive2://10.1.253.99:10000/default
Enter username:dmp
Enter password:dmp
show tables;
+--------------+
| tab_name |
+--------------+
| shaoaq_test |
+--------------+
select * from shaoaq_test;
+-----+
| id |
+-----+
+-----+

上传jdbc jar包

分发

启动

验证

5. hbase

配置

vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/regionservers
ocdata05
ocdata06
ocdata07
vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/backup-masters
ocdata08
vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/hbase-site.xml
<property>
<name>hbase.rootdir</name>
<value>hdfs://cdh5cluster/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ocdata09</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2183</value>
</property>
<property>
<name>hbase.regionserver.port</name>
<value>60328</value>
</property>
<property>
<name>hbase.regionserver.info.port</name>
<value>62131</value>
</property>
vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/hbase-env.sh
export JAVA_HOME=/home/cdh5/app/jdk1.7.0_51
export HBASE_CLASSPATH=/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop
export HBASE_HOME=/home/cdh5/app/hbase
export HADOOP_HOME=/home/cdh5/app/hadoop
export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
export HBASE_LIBRARY_PATH=${HBASE_HOME}/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HBASE_HOME}/lib/native
export HBASE_MANAGES_ZK=false
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/ all
./start-hbase.sh
./hbase shell
create 'hb_test', 'cf'
put 'hb_test','row1','cf:a','123'
get 'hb_test','row1'
COLUMN CELL
cf:a timestamp=1395204538429, value=123
1 row(s) in 0.0490 seconds
quit
./stop-hbase.sh

分发配置

启动

验证

停止

5. spark

spark当前可解压即用，yarn-client模式无需分发，只需修改客户端若干配置；

spark-1.1.0 on yarn的几个配置说明如下：

vim spark-env.sh
MASTER：部署模式，yarn-client/yarn-cluster/local
HADOOP_CONF_DIR：（必填）hadoop配置文件目录
SCALA_HOME：scala安装路径
SPARK_EXECUTOR_INSTANCES：spark申请的yarn worker总数
SPARK_EXECUTOR_CORES：每个worker申请的vcore数目
SPARK_EXECUTOR_MEMORY：每个worker申请的内存大小
SPARK_DRIVER_MEMORY：spark申请的appMaster内存大小
SPARK_YARN_APP_NAME：yarn中显示的spark任务名称
SPARK_YARN_QUEUE：spark任务队列
SPARK_SUBMIT_LIBRARY_PATH：spark任务执行时需要的库目录，如hadoop的native目录
SPARK_CLASSPATH：spark任务的classpath
SPARK_JAVA_OPTS：JVM进程参数，如gc类型、gc日志、dmp输出等
SPARK_HISTORY_OPTS：spark history-server配置参数，一般需要指定webUI端口、记录个数以及Event存储目录等
vim spark-defaults.conf
spark.local.dir：spark任务执行时的本地临时目录
spark.yarn.executor.memoryOverhead：每个worker的堆外内存大小，单位MB，yarn模式下需配置以防止内存溢出
spark.eventLog.enabled：是否记录Spark事件，用于应用程序在完成后重构webUI
spark.eventLog.dir：保存日志相关信息的路径，可以是hdfs://开头的HDFS路径，也可以是file://开头的本地路径，都需要提前创建
spark.eventLog.compress：是否压缩记录Spark事件，前提spark.eventLog.enabled为true，默认使用snappy

启停thrift-server：

使用spark-sql/thrift-server组件前需将hive-site.xml复制到$SPARK_HOME/conf目录下以使用hive的元数据和若干配置如server端口，可能需要去掉其中的一些多余或不支持的配置项，请留意；

$SPARK_HOME/sbin/start-thriftserver.sh

$SPARK_HOME/sbin/stop-thriftserver.sh

启停history-server：

$SPARK_HOME/sbin/start-history-server.sh
$SPARK_HOME/sbin/stop-history-server.sh

注意：

如hadoop中启用了lzo压缩需将hadoop-lzo-*.jar复制到SPARK_HOME/lib/目录下；
SPARK-1.1.0版本中spark-examples-*.jar关联的thrift版本与spark-assembly-*.jar不一致，需删除；

配置样例：

spark-env.sh

MASTER="yarn-client"

SPARK_HOME=/home/ochadoop/app/spark

SCALA_HOME=/home/ochadoop/app/scala

JAVA_HOME=/home/ochadoop/app/jdk

HADOOP_HOME=/home/ochadoop/app/hadoop

HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

SPARK_EXECUTOR_INSTANCES=50

SPARK_EXECUTOR_CORES=2

SPARK_EXECUTOR_MEMORY=4G

SPARK_DRIVER_MEMORY=3G

SPARK_YARN_APP_NAME="Spark-1.1.0"

#export SPARK_YARN_QUEUE="default"

SPARK_SUBMIT_LIBRARY_PATH=$SPARK_LIBRARY_PATH:$HADOOP_HOME/lib/native

SPARK_JAVA_OPTS="-verbose:gc -XX:-UseGCOverheadLimit -XX:+UseCompressedOops -XX:-PrintGCDetails -XX:+PrintGCTimeStamps $SPARK_JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ochadoop/app/spark/`date +%m%d%H%M%S`.hprof"

export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=1000 -Dspark.history.fs.logDirectory=hdfs://testcluster/eventLog"

spark-defaults.conf

spark.serializer org.apache.spark.serializer.KryoSerializer

spark.local.dir /data2/ochadoop/data/pseudo-dist/spark/local,/data3/ochadoop/data/pseudo-dist/spark/local

spark.io.compression.codec snappy

spark.speculation false

spark.yarn.executor.memoryOverhead 512

#spark.storage.memoryFraction 0.4

spark.eventLog.enabled true

spark.eventLog.dir hdfs://testcluster/eventLog

spark.eventLog.compress true

如下命令都是用root身份安装，或者在命令前加上sudo
采用yum安装方式安装
yum install mysql #安装mysql客户端
yum install mysql-server #安装mysql服务端
判断MySQL是否已经安装好：
chkconfig --list|grep mysql
启动mysql服务：
service mysqld start或者/etc/init.d/mysqld start
检查是否启动mysql服务：
/etc/init.d/mysqld status
设置MySQL开机启动：
chkconfig mysqld on
检查设置MySQL开机启动是否配置成功：
chkconfig --list|grep mysql
显示2 3 4 5为on
创建root管理员
mysqladmin -uroot password root
登录
mysql -uroot -proot