spark-1.6.0-cdh5.15.0 standalone 模式安装

 

更多资源

• github: https://github.com/opensourceteams/spark-scala-maven

• csdn(汇总视频在线看): https://blog.csdn.net/thinktothings/article/details/84726769

 

前置条件

• jdk ( 1.8.0_181)已安装

• scala(2.10.7)已安装

• hadoop(hadoop-2.6.0-cdh5.15.0)已安装,hdfs已启动

• 第三方jar

spark 依赖jar 文件配置，放置在 $SPARK_HOME/lib/*

​

parquet-hadoop-1.4.3.jar

jackson-databind-2.4.4.jar

jackson-annotations-2.4.4.jar

jackson-core-2.4.4.jar

jackson-module-scala_2.10-2.4.4.jar

​

​

<dependency>

<groupId>com.twitter</groupId>

<artifactId>parquet-hadoop</artifactId>

<version>1.4.3</version>

</dependency>

 

配置

 

环境变量配置

export JAVA_HOME=/opt/module/jdk/jdk1.8.0_191

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export SCALA_HOME=/opt/module/scala/scala-2.10.7

export HADOOP_HOME=/opt/module/bigdata/hadoop-2.6.0-cdh5.15.0

export SPARK_HOME=/opt/module/bigdata/spark-1.6.0-cdh5.15.0

​

export PATH=$JAVA_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

 

host配置

#/etc/hosts文件配置 IP到域名的配置

192.168.88.200 standalone.com standalone

 

spark-default.conf 配置

spark.master=spark://standalone.com:7077

spark.eventLog.enabled=true

spark.eventLog.dir=hdfs://standalone.com:9000/spark/log/eventLog

spark.serializer=org.apache.spark.serializer.KryoSerializer

spark.driver.memory=2g

​

​

#history

spark.history.fs.logDirectory=hdfs://standalone.com:9000/spark/log/historyEventLog

#The port to which the web interface of the history server binds.

spark.history.ui.port=18080

# The period at which information displayed by this history server is updated. Each update checks for any changes made to the event logs in persisted storage.

spark.history.fs.update.interval=10s

# The number of application UIs to retain. If this cap is exceeded, then the oldest applications will be removed.

spark.history.retainedApplications=50

spark.history.fs.cleaner.enabled=false

spark.history.fs.cleaner.interval=1d

spark.history.fs.cleaner.maxAge=7d

spark.history.ui.acls.enable=false

 

spark-env.sh 配置

export SPARK_DIST_CLASSPATH=${SPARK_HOME}/lib/*:$(${HADOOP_HOME}/bin/hadoop classpath)

SPARK_MASTER_IP=standalone.com

SPARK_MASTER_PORT=7077

SPARK_MASTER_WEBUI_PORT=8080

SPARK_WORKER_MEMORY=2g

​

SPARK_PRINT_LAUNCH_COMMAND=true #显示Main启动类执行的命令

 

salves配置

standalone.com

 

启动命令

 

启动 master 命令

start-master.sh

 

停止master命令

stop-master.sh

 

启动 worker 命令

start-slave.sh spark://standalone.com:7077

 

停止 worker 命令

stop-slave.sh

 

启动 history 命令

start-history-server.sh

 

停止 history 命令

stop-history-server.sh

 

启动 spark-shell 命令

spark-shell --master spark://standalone.com:7077

 

提交命令

 

spark-submit 命令

spark-submit \

--class com.opensource.bigdata.spark.standalone.RunTextFileMkString2 \

--master spark://standalone:7077 \

--executor-memory 1G \

--total-executor-cores 100 \

/root/temp/spark-scala-maven-1.0-SNAPSHOT.jar \

spark-submit \

--class com.opensource.bigdata.spark.standalone.RunTextFileMkString2 \

--master spark://standalone:7077 \

--deploy-mode client\

--executor-memory 1G \

--total-executor-cores 100 \

/root/temp/spark-scala-maven-1.0-SNAPSHOT.jar \