Spark2.0.0 Install And Examples
1.Scala 2.11.8 下载解压[root@sht-sgmhadoopnn-01 hadoop]# wget http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz[root@sht-sgmhadoopnn-01 hadoop]# tar xzvf scala-2.11.8.tgz[root@sht-sgmhadoopnn-01 hadoop]# mv scala-2.11.8 scala[root@sht-sgmhadoopnn-01 hadoop]# 2.将scala文件夹同步到集群其他机器[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopnn-02:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopdn-01:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopdn-02:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopdn-03:/hadoop/ 3.在集群的每台机器配置环境变量,生效###在文件末尾添加两行[root@sht-sgmhadoopnn-01 hadoop]# vi /etc/profileexport SCALA_HOME=/hadoop/scalaexport PATH=$SCALA_HOME/bin:$PATH [root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopnn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-01:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-03:/etc/profile [root@sht-sgmhadoopnn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopnn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-03 hadoop]# source /etc/profile ---------------------------------------------------------------------------------------------------------------------1.Spark2.0.0下载解压[root@sht-sgmhadoopnn-01 hadoop]# wget http://apache.website-solution.net/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz[root@sht-sgmhadoopnn-01 hadoop]# tar xzvf spark-2.0.0-bin-hadoop2.7.tgz[root@sht-sgmhadoopnn-01 hadoop]# mv spark-2.0.0-bin-hadoop2.7 spark 2.配置spark-env.sh[root@sht-sgmhadoopnn-01 conf]# pwd/hadoop/spark/conf[root@sht-sgmhadoopnn-01 conf]# cp spark-env.sh.template spark-env.sh[root@sht-sgmhadoopnn-01 conf]# ###添加以下5行[root@sht-sgmhadoopnn-01 conf]# vi spark-env.sh export SCALA_HOME=/hadoop/scalaexport JAVA_HOME=/usr/java/jdk1.7.0_67-clouderaexport SPARK_MASTER_IP=172.16.101.55export SPARK_WORKER_MEMORY=1gexport SPARK_PID_DIR=/hadoop/pidexport HADOOP_CONF_DIR=/hadoop/hadoop/etc/hadoop 3.配置slaves文件[root@sht-sgmhadoopnn-01 conf]# cp slaves.template slaves[root@sht-sgmhadoopnn-01 conf]# vi slaves sht-sgmhadoopdn-01sht-sgmhadoopdn-02sht-sgmhadoopdn-03 4.将spark文件夹copy到配置slaves文件的机器上[root@sht-sgmhadoopnn-01 hadoop]# scp -r spark root@sht-sgmhadoopdn-01:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r spark root@sht-sgmhadoopdn-02:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r spark root@sht-sgmhadoopdn-03:/hadoop/ 5.在集群的每台机器配置环境变量,生效[root@sht-sgmhadoopnn-01 hadoop]# vi /etc/profileexport SPARK_HOME=/hadoop/scalaexport PATH=$SPARK_HOME/bin:$PATH [root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopnn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-01:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-03:/etc/profile [root@sht-sgmhadoopnn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopnn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-03 hadoop]# source /etc/profile 6.启动spark[root@sht-sgmhadoopnn-01 sbin]# ./start-all.shstarting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-sht-sgmhadoopnn-01.outsht-sgmhadoopdn-01: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-sht-sgmhadoopdn-01.telenav.cn.outsht-sgmhadoopdn-02: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-sht-sgmhadoopdn-02.telenav.cn.outsht-sgmhadoopdn-03: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-sht-sgmhadoopdn-03.telenav.cn.out[root@sht-sgmhadoopnn-01 sbin]# 7.web查看http://sht-sgmhadoopnn-01:8080/ [root@sht-sgmhadoopnn-01 sbin]# jps27169 HMaster26233 NameNode26641 ResourceManager2312 Jps26542 DFSZKFailoverController2092 Master27303 RunJar26989 JobHistoryServer [root@sht-sgmhadoopdn-01 ~]# jps19907 Worker2086 jar17265 DataNode17486 NodeManager20055 Jps17377 JournalNode17697 HRegionServer3671 QuorumPeerMain 8.运行WordCount案例[root@sht-sgmhadoopnn-01 hadoop]# vi wordcount.txthello abc 123abc hadoop hello hdfsspark yarn123 abc hello hdfs spark wjp wjp abc hello [root@sht-sgmhadoopnn-01 bin]# spark-shellscala> scala> val textfile = sc.textFile("file:///hadoop/wordcount.txt")textfile: org.apache.spark.rdd.RDD[String] = file:///hadoop/wordcount.txt MapPartitionsRDD[1] at textFile at :24 scala> val count=textfile.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)count: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at :26 scala> count.collect()res0: Array[(String, Int)] = Array((hello,4), (123,2), (yarn,1), (abc,4), (wjp,2), (spark,2), (hadoop,1), (hdfs,2)) scala> ###val file=sc.textFile("hadoop fs -ls hdfs://172.16.101.56:8020/wordcount.txt") val file = sc.textFile("hdfs://namenode:8020/path/to/input")val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _)counts.saveAsTextFile("hdfs://namenode:8020/output") --------------------------------------------------------------------------------------------------------------------------------------------------------a.本地模式两线程运行#[root@sht-sgmhadoopdn-01 ~]# ./bin/run-example SparkPi 2>&1 | grep "Pi is roughly" [root@sht-sgmhadoopnn-01 spark]# ./bin/run-example SparkPi 10 --master local[2] b.Spark Standalone 集群模式运行[root@sht-sgmhadoopnn-01 spark]# ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \--master spark://sht-sgmhadoopnn-01:7077 \examples/jars/spark-examples_2.11-2.0.0.jar \100 c.注意 Spark on YARN 支持两种运行模式,分别为yarn-cluster和yarn-client,具体的区别可以看这篇博文,从广义上讲,yarn-cluster适用于生产环境;而yarn-client适用于交互和调试,也就是希望快速地看到application的输出。 #Spark on YARN 集群上 yarn-cluster 模式运行[root@sht-sgmhadoopnn-01 spark]# ./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master yarn-cluster ./examples/jars/spark-examples_2.11-2.0.0.jar \10 spark-submit \--class org.apache.spark.examples.SparkPi \--master yarn-cluster \--num-executors 3 \--driver-memory 4g \--executor-memory 2g \--executor-cores 1 \$SPARK_HOME/examples/jars/spark-examples_2.11-2.0.0.jar \10