首页 文章 精选 留言 我的

精选列表

搜索[异常],共10006篇文章
优秀的个人博客,低调大师

spark on yarn 配置及异常解决

版权声明:本文由董可伦首发于https://dongkelun.com,非商业转载请注明作者及原创出处。商业转载请联系作者本人。 https://blog.csdn.net/dkl12/article/details/80256461 我的原创地址:https://dongkelun.com/2018/04/16/sparkOnYarnConf/ 前言 YARN 是在Hadoop 2.0 中引入的集群管理器,它可以让多种数据处理框架运行在一个共享的资源池上,并且通常安装在与Hadoop 文件系统(简称HDFS)相同的物理节点上。在这样配置的YARN 集群上运行Spark 是很有意义的,它可以让Spark 在存储数据的物理节点上运行,以快速访问HDFS 中的数据。 1、配置 1.1 配置HADOOP_CONF_DIR vim /etc/profile export HADOOP_CONF_DIR=/opt/hadoop-2.7.5/etc/hadoop source /etc/profile 1.2 命令行启动 spark-shell --master yarn 但是在spark2.x里会报一个错误 18/04/16 07:59:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/04/16 07:59:27 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 18/04/16 07:59:54 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101) at $line3.$read$$iw$$iw.<init>(<console>:15) at $line3.$read$$iw.<init>(<console>:42) at $line3.$read.<init>(<console>:44) at $line3.$read$.<init>(<console>:48) at $line3.$read$.<clinit>(<console>) at $line3.$eval$.$print$lzycompute(<console>:7) at $line3.$eval$.$print(<console>:6) at $line3.$eval.$print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786) at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047) at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638) at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637) at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31) at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19) at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569) at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565) at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807) at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681) at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395) at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38) at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37) at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214) at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37) at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:98) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909) at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97) at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909) at org.apache.spark.repl.Main$.doMain(Main.scala:74) at org.apache.spark.repl.Main$.main(Main.scala:54) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18/04/16 07:59:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered! 18/04/16 07:59:54 WARN MetricsSystem: Stopping a MetricsSystem that is not running org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:101) ... 47 elided <console>:14: error: not found: value spark import spark.implicits._ ^ <console>:14: error: not found: value spark import spark.sql ^ Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.2.1 /_/ Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45) Type in expressions to have them evaluated. Type :help for more information. scala> 2、错误解决 2.1 添加spark.yarn.jars 首先看到第二条warn Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 联想到是不是这条warn信息导致的,然后根据这条warn信息上网查了一下,再根据错误信息也查了一下 Yarn application has already ended! It might have been killed or unable to ... 发现,都是说要配置spark.yarn.jars,于是按照如下命令配置 hdfs dfs -mkdir /hadoop hdfs dfs -mkdir /hadoop/spark_jars hdfs dfs -put /opt/spark-2.2.1-bin-hadoop2.7/jars/* /hadoop/spark_jars cd /opt/spark-2.2.1-bin-hadoop2.7/conf/ cp spark-defaults.conf.template spark-defaults.conf vim spark-defaults.conf 在最下面添加: spark.yarn.jars hdfs://192.168.44.128:8888/hadoop/spark_jars/* (注意后面的*不能去掉) 然后启动spark-shell,发现还是报相似错误(没了warn) 2.2 配置hadoop的yarn-site.xml 因为java8导致的问题 vim /opt/hadoop-2.7.5/etc/hadoop/yarn-site.xml 添加: <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> 再次启动spark-shell,成功! 3、意外之喜 由于要写博客记录,所以需要将错误还原,第一次只将spark.yarn.jars注释掉,启动spark-shell,发现是成功的,只是会有条warn而已,也就是说,这个错误的根本原因,是java8导致没有配置2.2中的yarn-site.xml!! 参考资料 https://blog.csdn.net/lxhandlbb/article/details/54410644 https://blog.csdn.net/gg584741/article/details/72825713

优秀的个人博客,低调大师

Intellij IDEA创建Android项目异常

一、今天在使用IDEA创建Android项目的时候,一直卡在这个位置,无论填什么就是无法继续 二、然后我就退出了,发现IDEA给我提示了错误 null java.lang.NullPointerException at com.android.tools.idea.sdk.wizard.InstallSelectedPackagesStep.onEntering(InstallSelectedPackagesStep.java:143) at com.android.tools.idea.wizard.model.ModelWizard.goForward(ModelWizard.java:303) at com.android.tools.idea.npw.ideahost.IdeaWizardAdapter.doNextAction(IdeaWizardAdapter.java:88) at com.android.tools.idea.npw.ideahost.AndroidModuleBuilder.doNextAction(AndroidModuleBuilder.java:138) at com.intellij.ide.util.newProjectWizard.AbstractProjectWizard.doNextAction(AbstractProjectWizard.java:260) at com.intellij.ide.wizard.AbstractWizard$5.actionPerformed(AbstractWizard.java:199) at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022) 三、在IDEA报错后,点击IDEA界面的问号,即可看到问题详情 四、原来是IDEA没有配置Android的SDK。 将安卓SDK配置下即可解决问题了。

优秀的个人博客,低调大师

Hadoop的NameNode无法启动异常

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2014 - 09 - 20 08 : 08 : 54 , 179 ERRORorg.apache.hadoop.hdfs.server.namenode.FSNamesystem:FSNamesysteminitializationfailed. java.io.IOException:NameNodeisnotformatted. atorg.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java: 331 ) atorg.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java: 104 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java: 427 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java: 395 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java: 299 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java: 569 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java: 1479 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 1488 ) 2014 - 09 - 20 08 : 08 : 54 , 199 ERRORorg.apache.hadoop.hdfs.server.namenode.NameNode:java.io.IOException:NameNodeisnotformatted. atorg.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java: 331 ) atorg.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java: 104 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java: 427 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java: 395 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java: 299 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java: 569 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java: 1479 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 1488 ) 解决方法: 格式化NameNode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 [root@hadoopbin] #./hadoopnamenode-format 13 /06/28 22:50:23INFOnamenode.NameNode:STARTUP_MSG: /************************************************************ STARTUP_MSG:StartingNameNode STARTUP_MSG:host=java.net.UnknownHostException:hadoop:hadoop STARTUP_MSG:args=[- format ] STARTUP_MSG:version=0.20.2 STARTUP_MSG:build=https: //svn .apache.org /repos/asf/hadoop/common/branches/branch-0 .20-r911707;compiledby 'chrisdo' onFriFeb1908:07:34UTC2010 ************************************************************/ 13 /06/28 22:50:23INFOnamenode.FSNamesystem:fsOwner=root,root 13 /06/28 22:50:23INFOnamenode.FSNamesystem:supergroup=supergroup 13 /06/28 22:50:23INFOnamenode.FSNamesystem:isPermissionEnabled= true 13 /06/28 22:50:38INFOmetrics.MetricsUtil:UnabletoobtainhostName java.net.UnknownHostException:hadoop:hadoop atjava.net.InetAddress.getLocalHost(InetAddress.java:1354) atorg.apache.hadoop.metrics.MetricsUtil.getHostName(MetricsUtil.java:91) atorg.apache.hadoop.metrics.MetricsUtil.createRecord(MetricsUtil.java:80) atorg.apache.hadoop.hdfs.server.namenode.FSDirectory.initialize(FSDirectory.java:73) atorg.apache.hadoop.hdfs.server.namenode.FSDirectory.<init>(FSDirectory.java:68) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:379) atorg.apache.hadoop.hdfs.server.namenode.NameNode. format (NameNode.java:854) atorg.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948) atorg.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) 13 /06/28 22:50:38INFOcommon.Storage:Image file ofsize94saved in 0seconds. 13 /06/28 22:50:38INFOcommon.Storage:Storagedirectory /tmp/hadoop-root/dfs/name hasbeensuccessfullyformatted. 13 /06/28 22:50:38INFOnamenode.NameNode:SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG:ShuttingdownNameNodeatjava.net.UnknownHostException:hadoop:hadoop ************************************************************/ 问题: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 2014 - 09 - 20 08 : 02 : 32 , 476 INFOorg.apache.hadoop.hdfs.server.common.Storage:Storagedirectory/tmp/hadoop-root/dfs/namedoesnotexist 2014 - 09 - 20 08 : 02 : 32 , 480 ERRORorg.apache.hadoop.hdfs.server.namenode.FSNamesystem:FSNamesysteminitializationfailed. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory/tmp/hadoop-root/dfs/nameisinaninconsistentstate:storagedirectorydoesnotexistorisnotaccessible. atorg.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java: 304 ) atorg.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java: 104 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java: 427 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java: 395 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java: 299 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java: 569 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java: 1479 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 1488 ) 2014 - 09 - 20 08 : 02 : 32 , 494 ERRORorg.apache.hadoop.hdfs.server.namenode.NameNode:org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory/tmp/hadoop-root/dfs/nameisinaninconsistentstate:storagedirectorydoesnotexistorisnotaccessible. atorg.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java: 304 ) atorg.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java: 104 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java: 427 ) atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java: 395 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java: 299 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java: 569 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java: 1479 ) atorg.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 1488 ) 2014 - 09 - 20 08 : 02 : 32 , 495 INFOorg.apache.hadoop.hdfs.server.namenode.NameNode:SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG:ShuttingdownNameNodeatlocalhost/127.0.0.1 ************************************************************/ 解决: 1 mkidr-pv /tmp/hadoop-root/dfs/name 本文转自 梦朝思夕 51CTO博客,原文链接:http://blog.51cto.com/qiangmzsx/1556336

资源下载

更多资源
腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring

Spring框架(Spring Framework)是由Rod Johnson于2002年提出的开源Java企业级应用框架,旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念,提供核心容器、应用上下文、数据访问集成等模块,支持整合Hibernate、Struts等第三方框架,其适用范围不仅限于服务器端开发,绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能,例如代码缩略图,Python的插件,代码段等。还可自定义键绑定,菜单和工具栏。Sublime Text 的主要功能包括:拼写检查,书签,完整的 Python API , Goto 功能,即时项目切换,多选择,多窗口等等。Sublime Text 是一个跨平台的编辑器,同时支持Windows、Linux、Mac OS X等操作系统。

用户登录
用户注册