hadoop-spark-hive-hbase配置相关说明-低调大师

hadoop-spark-hive-hbase配置相关说明

2015-10-09 820

1. zookeeper

配置

cp app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo_sample.cfg app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg
vim app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg
dataDir=/home/cdh5/tmp/zookeeper
clientPort=2183
server.1=ocdata09:2888:3888
mkdir -p /home/cdh5/tmp/zookeeper
vim /home/cdh5/tmp/zookeeper/myid
echo "1" > /home/cdh5/tmp/zookeeper/myid

初始化操作：

或者 ./runRemoteCmd.sh '~/och200/zookeeper/bin/zkServer-initialize.sh --myid=1' zoo

分发配置

./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/zoo.cfg app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/ zoo
./runRemoteCmd.sh "app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/zkServer.sh start" zoo
./runRemoteCmd.sh 'echo ruok | nc localhost 2183' zoo
./runRemoteCmd.sh "app/ochadoop-och3.0.0-SNAPSHOT/zookeeper-3.4.5-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/zkServer.sh stop" zoo

启动

验证

停止

2. HDFS

配置hadoop
hdfs-site.xml

<property>
<name>dfs.nameservices</name>
<value>cdh5cluster</value>
<description>
Comma-separated list of nameservices.
</description>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:50011</value>
<description>
The datanode server address and port for data transfer.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:50076</value>
<description>
The datanode http server address and port.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.datanode.ipc.address</name>
<value>0.0.0.0:50021</value>
<description>
The datanode ipc server address and port.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
（）
<name>dfs.nameservices</name>
<value>cdh5cluster</value>
</property>
<property>
（命名空间中所有NameNode的唯一标示名称。可以配置多个，使用逗号分隔。该名称是可以让DataNode知道每个集群的所有NameNode.当前，每个集群最多只能配置两个NameNode）
<name>dfs.ha.namenodes.cdh5cluster</name>
<value>nn1,nn2</value>
<description></description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data1/cdh5/dfs/name</value>
<description>Determines where on the local filesystem the DFS name node should store the name table.If this is a comma-delimited list of directories,then name table is replicated in all of the directories,for redundancy.</description>
<final>true</final>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data1/cdh5/dfs/data,file:///data2/cdh5/dfs/data,file:///data3/cdh5/dfs/data</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permission</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
<description>
Boolean which enables backend datanode-side support for the experimental DistributedFileSystem*getFileVBlockStorageLocations API.
</description>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
<description>
If "true", enable permission checking in HDFS.
If "false", permission checking is turned off,
but all other behavior is unchanged.
Switching from one parameter value to the other does not change the mode,
owner or group of files or directories.
</description>
</property>
<property>
(每个NAMENODE监听的RPC地址)
<name>dfs.namenode.rpc-address.cdh5cluster.nn1</name>
<value>ocdata09:8030</value>
<description>节点NN1的RPC地址</description>
</property>
<property>
<name>dfs.namenode.rpc-address.cdh5cluster.nn2</name>
<value>ocdata08:8030</value>
<description>节点NN2的RPC地址</description>
</property>
<property>
<name>dfs.namenode.http-address.cdh5cluster.nn1</name>
<value>ocdata09:50082</value>
<description>节点NN1的HTTP地址</description>
</property>
<property>
<name>dfs.namenode.http-address.cdh5cluster.nn2</name>
<value>ocdata08:50082</value>
<description>节点NN2的HTTP地址</description>
</property>
<property>
（这是NameNode读写JNs的uri。通过这个uri，NameNodes可以读写edit log内容。URI的格式”qjournal://host1:port1;host2:port2;host3:port3/journalId”。这里的host1、host2、host3指的是Journal Node的地址，这里必须是奇数个，至少3个;其中journaId是集群的唯一标示符，对于多个联邦命名空间，也使用同一个journaId。配置如下J
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://ocdata05:8488;ocdata06:8488;ocdata07:8488/cdh5cluster</value>
<description>采用3个journalnode节点存储元数据，这是IP与端口</description>
</property>

<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/cdh5/journaldata/jn</value>
<description>journaldata的存储路径</description>
</property>
<property>
<name>dfs.journalnode.rpc-address</name>
<value>0.0.0.0:8488</value>
</property>
<property>
<name>dfs.journalnode.http-address</name>
<value>0.0.0.0:8483</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.cdh5cluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
<description>该类用来判断哪个namenode处于生效状态</description>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>shell(/bin/true)</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>10000</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
<description>
Whether automatic failover is enabled. See the HDFS High
Availability documentation for details on automatic HA
configuration.
</description>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>ocdata09:2183</value>
<description>1个zookeeper节点</description>
</property>
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>4096</value>
<description>
Specifies the maximum number of threads to use for transferring data
in and out of the DN.
</description>
</property>
<property>
<name>dfs.blocksize</name>
<value>64m</value>
<description>
The default block size for new files, in bytes.
You can use the following suffix (case insensitive):
k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),
Or provide complete size in bytes (such as 134217728 for 128 MB).
</description>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>20</value>
<description>The number of server threads for the namenode.</description>
</property>
<property>
（这是一个描述集群中NameNode节点的URI(包括协议、主机名称\端口号—)集群里面的每一台机器都要知道NAMENODE的地址。DataNode节点会先在NAMENODE上注册，这样它们的数据才可以被使用。独立的客户端程序通过这个URI跟DATANODE交互，以取得文件的块列表。）
<name>fs.defaultFS</name>
<value>hdfs://cdh5cluster</value>
</property>
<property>
（hadoop文件系统依赖的基础配置，很多路径都依赖它。如果hdfs-site.xml中不配置namenode和datanode的存放位置，默认就放在这个路径中）
<name>hadoop.tmp.dir</name>
<value>/home/cdh5/tmp/hadoop/hadoop-${user.name}</value>
</property>
<property>

core-site.xml

(对本地jar包进行加载)

<name>io.native.lib.available</name>
<value>true</value>
<description>Should native hadoop libraries, if present, be used.</description>
</property>
(压缩和解压编码类列表，用逗号分隔，这些类是使用java ServiceLoader加载，如果不设置就为null)
<property>
<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
ocdata05
ocdata06
ocdata07
ocdata08
ocdata09
ocdata05
ocdata06
export JAVA_HOME=/home/cdh5/app/jdk1.7.0_21
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc all

slaves

masters

hadoop-env.sh

分发

初始化HDFS：

主节点执行

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs zkfc -formatZK

./runRemoteCmd.sh 'app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh start journalnode' jn

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs namenode -format -initializeSharedEdits

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh start namenode

备节点执行

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hdfs namenode -bootstrapStandby

完成

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh stop namenode

./runRemoteCmd.sh 'app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/hadoop-daemon.sh stop journalnode' jn

启动HDFS

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/start-dfs.sh
http://10.1.253.99:50082/dfshealth.html (active)
http://10.1.253.98:50082/dfshealth.html (standby)
http://10.1.253.97:8483/journalstatus.jsp
http://10.1.253.96:8483/journalstatus.jsp
http://10.1.253.95:8483/journalstatus.jsp
app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/stop-dfs.sh

验证：

停止HDFS

3. Yarn

配置YARN

mapred-site.xml

cp app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml.template app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml
vim app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>8350</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>0.0.0.0:10121</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>0.0.0.0:19868</value>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>0.0.0.0:50330</value>
</property>
<property>
<name>mapreduce.tasktracker.http.address</name>
<value>0.0.0.0:50360</value>
</property>
vim app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop/yarn-site.xml
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-rm-cluster</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<description>Id of the current ResourceManager. Must be set explicitly on each ResourceManager to the appropriate value.</description>
<name>yarn.resourcemanager.ha.id</name>
<value>rm1</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>ocdata09:2183</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>ocdata09:2183</value>
</property>
<property>
<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
<value>5000</value>
</property>
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>ocdata08:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>ocdata08:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>ocdata08:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>ocdata08:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>ocdata08:23141</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm1</name>
<value>ocdata08:23142</value>
</property>
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>ocdata09:23140</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>ocdata09:23130</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>ocdata09:23188</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>ocdata09:23125</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>ocdata09:23141</value>
</property>
<property>
<name>yarn.resourcemanager.ha.admin.address.rm2</name>
<value>ocdata09:23142</value>
</property>
<property>
<description>Address where the localizer IPC is.</description>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:23344</value>
</property>
<property>
<description>NM Webapp address.</description>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:23999</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/tmp/pseudo-dist/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/tmp/pseudo-dist/yarn/log</value>
</property>
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc all
app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/start-yarn.sh

yarn-site.xml

分发

Yarn的启动停止 YARN不需要初始化，登录主节点执行

cdh5 yarn的ha需要手动启动备节点

./runRemoteCmd.sh "cd app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin; ./yarn-daemon.sh start resourcemanager" rm2

验证

http://10.1.253.98:23188/cluster (有节点列表，active)

http://10.1.253.99:23188/cluster (无节点列表，standby)

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/bin/hadoop jar app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar TestDFSIO -write -nrFiles 40 -fileSize 20MB

停止YARN

app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin/stop-yarn.sh

手动停止备节点

./runRemoteCmd.sh "cd app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/sbin; ./yarn-daemon.sh stop resourcemanager" rm2

4. hive

配置

cp hive-env.sh.template hive-env.sh
vim hive-env.sh
export HADOOP_HOME=/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT
cp hive-default.xml.template hive-site.xml
vim hive-site.xml

删除其他配置项，只保留：

（配置元数据库,一般为mysql）

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://10.1.252.69:3306/cdh5?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=UTF-8</value>

</property>

(配置元数据库的Driver)

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

<description>Driver class name for a JDBC metastore</description>

</property>

(配置元数据库的名称)

<name>javax.jdo.option.ConnectionUserName</name>

<description>username to use against metastore database</description>

</property>

(配置元数据库的密码)

<name>javax.jdo.option.ConnectionPassword</name>

<description>password to use against metastore database</description>

</property>

元数据库配置

CREATE USER cdh5 IDENTIFIED BY 'cdh5';
CREATE DATABASE cdh5;
alter database cdh5 character set latin1;
grant all privileges on *.* to cdh5@"%" identified by "cdh5";
flush privileges;
scp mysql-connector-java-5.1.26.jar cdh5@10.1.253.99:/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hive-0.12.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/lib/
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hive-0.12.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT app/ochadoop-och3.0.0-SNAPSHOT/ hive
nohup ./hiveserver2 &
jdbc:
jdbc:hive2://10.1.253.99:10000/default
org.apache.hive.jdbc.HiveDriver
lib: Hadoop和hive下所有jar包
!connect jdbc:hive2://10.1.253.99:10000/default
Enter username:dmp
Enter password:dmp
show tables;
+--------------+
| tab_name |
+--------------+
| shaoaq_test |
+--------------+
select * from shaoaq_test;
+-----+
| id |
+-----+
+-----+

上传jdbc jar包

分发

启动

验证

5. hbase

配置

vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/regionservers
ocdata05
ocdata06
ocdata07
vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/backup-masters
ocdata08
vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/hbase-site.xml
<property>
<name>hbase.rootdir</name>
<value>hdfs://cdh5cluster/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ocdata09</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2183</value>
</property>
<property>
<name>hbase.regionserver.port</name>
<value>60328</value>
</property>
<property>
<name>hbase.regionserver.info.port</name>
<value>62131</value>
</property>
vim app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf/hbase-env.sh
export JAVA_HOME=/home/cdh5/app/jdk1.7.0_51
export HBASE_CLASSPATH=/home/cdh5/app/ochadoop-och3.0.0-SNAPSHOT/hadoop-2.2.0-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/etc/hadoop
export HBASE_HOME=/home/cdh5/app/hbase
export HADOOP_HOME=/home/cdh5/app/hadoop
export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
export HBASE_LIBRARY_PATH=${HBASE_HOME}/lib/native
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HBASE_HOME}/lib/native
export HBASE_MANAGES_ZK=false
./deploy.sh app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/conf app/ochadoop-och3.0.0-SNAPSHOT/hbase-0.96.1.1-cdh5.0.0-beta-2-och3.0.0-SNAPSHOT/ all
./start-hbase.sh
./hbase shell
create 'hb_test', 'cf'
put 'hb_test','row1','cf:a','123'
get 'hb_test','row1'
COLUMN CELL
cf:a timestamp=1395204538429, value=123
1 row(s) in 0.0490 seconds
quit
./stop-hbase.sh

分发配置

启动

验证

停止

5. spark

spark当前可解压即用，yarn-client模式无需分发，只需修改客户端若干配置；

spark-1.1.0 on yarn的几个配置说明如下：

vim spark-env.sh
MASTER：部署模式，yarn-client/yarn-cluster/local
HADOOP_CONF_DIR：（必填）hadoop配置文件目录
SCALA_HOME：scala安装路径
SPARK_EXECUTOR_INSTANCES：spark申请的yarn worker总数
SPARK_EXECUTOR_CORES：每个worker申请的vcore数目
SPARK_EXECUTOR_MEMORY：每个worker申请的内存大小
SPARK_DRIVER_MEMORY：spark申请的appMaster内存大小
SPARK_YARN_APP_NAME：yarn中显示的spark任务名称
SPARK_YARN_QUEUE：spark任务队列
SPARK_SUBMIT_LIBRARY_PATH：spark任务执行时需要的库目录，如hadoop的native目录
SPARK_CLASSPATH：spark任务的classpath
SPARK_JAVA_OPTS：JVM进程参数，如gc类型、gc日志、dmp输出等
SPARK_HISTORY_OPTS：spark history-server配置参数，一般需要指定webUI端口、记录个数以及Event存储目录等
vim spark-defaults.conf
spark.local.dir：spark任务执行时的本地临时目录
spark.yarn.executor.memoryOverhead：每个worker的堆外内存大小，单位MB，yarn模式下需配置以防止内存溢出
spark.eventLog.enabled：是否记录Spark事件，用于应用程序在完成后重构webUI
spark.eventLog.dir：保存日志相关信息的路径，可以是hdfs://开头的HDFS路径，也可以是file://开头的本地路径，都需要提前创建
spark.eventLog.compress：是否压缩记录Spark事件，前提spark.eventLog.enabled为true，默认使用snappy

启停thrift-server：

使用spark-sql/thrift-server组件前需将hive-site.xml复制到$SPARK_HOME/conf目录下以使用hive的元数据和若干配置如server端口，可能需要去掉其中的一些多余或不支持的配置项，请留意；

$SPARK_HOME/sbin/start-thriftserver.sh

$SPARK_HOME/sbin/stop-thriftserver.sh

启停history-server：

$SPARK_HOME/sbin/start-history-server.sh
$SPARK_HOME/sbin/stop-history-server.sh

注意：

如hadoop中启用了lzo压缩需将hadoop-lzo-*.jar复制到SPARK_HOME/lib/目录下；
SPARK-1.1.0版本中spark-examples-*.jar关联的thrift版本与spark-assembly-*.jar不一致，需删除；

配置样例：

spark-env.sh

MASTER="yarn-client"

SPARK_HOME=/home/ochadoop/app/spark

SCALA_HOME=/home/ochadoop/app/scala

JAVA_HOME=/home/ochadoop/app/jdk

HADOOP_HOME=/home/ochadoop/app/hadoop

HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

SPARK_EXECUTOR_INSTANCES=50

SPARK_EXECUTOR_CORES=2

SPARK_EXECUTOR_MEMORY=4G

SPARK_DRIVER_MEMORY=3G

SPARK_YARN_APP_NAME="Spark-1.1.0"

#export SPARK_YARN_QUEUE="default"

SPARK_SUBMIT_LIBRARY_PATH=$SPARK_LIBRARY_PATH:$HADOOP_HOME/lib/native

SPARK_JAVA_OPTS="-verbose:gc -XX:-UseGCOverheadLimit -XX:+UseCompressedOops -XX:-PrintGCDetails -XX:+PrintGCTimeStamps $SPARK_JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ochadoop/app/spark/`date +%m%d%H%M%S`.hprof"

export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=1000 -Dspark.history.fs.logDirectory=hdfs://testcluster/eventLog"

spark-defaults.conf

spark.serializer org.apache.spark.serializer.KryoSerializer

spark.local.dir /data2/ochadoop/data/pseudo-dist/spark/local,/data3/ochadoop/data/pseudo-dist/spark/local

spark.io.compression.codec snappy

spark.speculation false

spark.yarn.executor.memoryOverhead 512

#spark.storage.memoryFraction 0.4

spark.eventLog.enabled true

spark.eventLog.dir hdfs://testcluster/eventLog

spark.eventLog.compress true

如下命令都是用root身份安装，或者在命令前加上sudo
采用yum安装方式安装
yum install mysql #安装mysql客户端
yum install mysql-server #安装mysql服务端
判断MySQL是否已经安装好：
chkconfig --list|grep mysql
启动mysql服务：
service mysqld start或者/etc/init.d/mysqld start
检查是否启动mysql服务：
/etc/init.d/mysqld status
设置MySQL开机启动：
chkconfig mysqld on
检查设置MySQL开机启动是否配置成功：
chkconfig --list|grep mysql
显示2 3 4 5为on
创建root管理员
mysqladmin -uroot password root
登录
mysql -uroot -proot

微信关注我们

原文链接：https://yq.aliyun.com/articles/609179

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

Shuffle相关分析

Shuffle描述是一个过程，表现出的是多对多的依赖关系。Shuffle是连接map阶段和Reduce阶段的纽带，每个Reduce Task都会从Map Task产生的数据里读取其中的一片数据。Shuffle通常分为两个部分:Map阶段的数据准备和Reduce阶段的数据副本。 Map阶段根据Reduce阶段的Task数量来决定每个Map Task输出的数据分片的个数，这些数据分片可能保存在内存中或者磁盘上，这些分片的存在形式可能是每个分片一个文件，也可能是多个分片放在一个数据文件中，外加一个索引来记录每个分片在数据文件中的偏移量。（RDD中的窄依赖除外，恰好是一对一的） 1、 Shuffle写 Spark中Shuffle输出的ShuffleMapTask会为每个ResultTask创建对应的Bucket,ShuffleMapTask产生的结果会根据设置的partitionner得到对应的BucketId.然后填充到对应的Bucket中去，所以每个ShuffleMapTask创建Bucket的数据是和ResultTask的数目相等的。 ShuffleMapTask创建的Bucket对...

2015-10-10

709

import org.apache.spark.{SparkContext, SparkConf}import org.apache.spark.sql.{SaveMode, DataFrame}import scala.collection.mutable.ArrayBufferimport main.asiainfo.coc.tools.Configureimport org.apache.spark.sql.hive.HiveContextimport java.sql.DriverManagerimport java.sql.Connection 1 连接前台数据源查询前台MYSQL中的数据 val DIM_COC_INDEX_INFO_DDL = s""" CREATE TEMPORARY TABLE DIM_COC_INDEX_INFO USING org.apache.spark.sql.jdbc OPTIONS ( url '${mySQLUrl}', dbtable 'DIM_COC_INDEX_INFO' )""".stripMargin sqlContext.s...

2015-10-10

596

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。