Hadoop HDFS： the directory item limit is exceed: limit=1048576问题的解决-低调大师

Hadoop HDFS： the directory item limit is exceed: limit=1048576问题的解决

2017-11-06 721

问题描述：

1.文件无法写入hadoop hdfs文件系统；

2.hadoop namenode日志记录

 
        the directory item limit is exceed: limit=1048576

3.hadoop单个目录下文件超1048576个，默认limit限制数为1048576，所以要调大limit限制数

解决办法：

 
        hdfs-site.xml配置文件添加配置参数
       
        <property>
       
        <name>dfs.namenode.fs-limits.max-directory-items<
        /name
        > 
       
        <value>3200000<
        /value
        > 
       
        <description>Defines the maximum number of items that a directory may 
       
        contain. Cannot 
        set 
        the property to a value 
        less 
        than 1 or 
        more 
        than 
       
        6400000.<
        /description
        > 
       
        <
        /property
        > 
       
        将配置文件推送至hadoop集群所有节点
       
        重启hadoop服务

延伸问题：

 
  
    
      
     
        2017-11-17 13:03:31,795 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
       

        2017-11-17 13:03:31,797 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
       

        /************************************************************
       
 
        SHUTDOWN_MSG: Shutting down NameNode at name-01
        /10
        .0.0.101 
       

        ************************************************************/
       

        2017-11-17 13:09:45,016 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: 
       

        /************************************************************
       

        STARTUP_MSG: Starting NameNode
       
 
        STARTUP_MSG:   host = name-01
        /10
        .0.0.101 
       

        STARTUP_MSG:   args = []
       

        STARTUP_MSG:   version = 2.6.0-cdh5.7.4
       
 
        STARTUP_MSG:   classpath = 
        /etc/hadoop/conf
        :
        /usr/lib/hadoop/lib/commons-io-2
        .4.jar:
        /usr/lib/hadoop/lib/jettison-1
        .1.jar:
        /usr/lib/hadoop/lib/stax-api-1
        .0-2.jar:
        /usr/lib/hadoop/lib/zookeeper
        .jar:
        /usr/lib/hadoop/lib/avro
        .jar:
        /usr/lib/hadoop/lib/jsch-0
        .1.42.jar:
        /usr/lib/hadoop/lib/java-xmlbuilder-0
        .4.jar:
        /usr/lib/hadoop/lib/curator-client-2
        .7.1.jar:
        /usr/lib/hadoop/lib/jasper-runtime-5
        .5.23.jar:
        /usr/lib/hadoop/lib/servlet-api-2
        .5.jar:
        /usr/lib/hadoop/lib/hamcrest-core-1
        .3.jar:
        /usr/lib/hadoop/lib/commons-net-3
        .1.jar:
        /usr/lib/hadoop/lib/jets3t-0
        .9.0.jar:
        /usr/lib/hadoop/lib/httpcore-4
        .2.5.jar:
        /usr/lib/hadoop/lib/jersey-server-1
        .9.jar:
        /usr/lib/hadoop/lib/activation-1
        .1.jar:
        /usr/lib/hadoop/lib/slf4j-api-1
        .7.5.jar:
        /usr/lib/hadoop/lib/apacheds-kerberos-codec-2
        .0.0-M15.jar:
        /usr/lib/hadoop/lib/xmlenc-0
        .52.jar:
        /usr/lib/hadoop/lib/commons-beanutils-core-1
        .8.0.jar:
        /usr/lib/hadoop/lib/commons-codec-1
        .4.jar:
        /usr/lib/hadoop/lib/jaxb-api-2
        .2.2.jar:
        /usr/lib/hadoop/lib/curator-recipes-2
        .7.1.jar:
        /usr/lib/hadoop/lib/jsp-api-2
        .1.jar:
        /usr/lib/hadoop/lib/aws-java-sdk-s3-1
        .10.6.jar:
        /usr/lib/hadoop/lib/apacheds-i18n-2
        .0.0-M15.jar:
        /usr/lib/hadoop/lib/jaxb-impl-2
        .2.3-1.jar:
        /usr/lib/hadoop/lib/log4j-1
        .2.17.jar:
        /usr/lib/hadoop/lib/commons-logging-1
        .1.3.jar:
        /usr/lib/hadoop/lib/jetty-util-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop/lib/jackson-xc-1
        .8.8.jar:
        /usr/lib/hadoop/lib/commons-math3-3
        .1.1.jar:
        /usr/lib/hadoop/lib/jackson-jaxrs-1
        .8.8.jar:
        /usr/lib/hadoop/lib/jetty-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop/lib/htrace-core4-4
        .0.1-incubating.jar:
        /usr/lib/hadoop/lib/junit-4
        .11.jar:
        /usr/lib/hadoop/lib/logredactor-1
        .0.3.jar:
        /usr/lib/hadoop/lib/api-util-1
        .0.0-M20.jar:
        /usr/lib/hadoop/lib/commons-httpclient-3
        .1.jar:
        /usr/lib/hadoop/lib/jersey-json-1
        .9.jar:
        /usr/lib/hadoop/lib/api-asn1-api-1
        .0.0-M20.jar:
        /usr/lib/hadoop/lib/commons-cli-1
        .2.jar:
        /usr/lib/hadoop/lib/aws-java-sdk-core-1
        .10.6.jar:
        /usr/lib/hadoop/lib/commons-configuration-1
        .6.jar:
        /usr/lib/hadoop/lib/mockito-all-1
        .8.5.jar:
        /usr/lib/hadoop/lib/xz-1
        .0.jar:
        /usr/lib/hadoop/lib/aws-java-sdk-kms-1
        .10.6.jar:
        /usr/lib/hadoop/lib/snappy-java-1
        .0.4.1.jar:
        /usr/lib/hadoop/lib/guava-11
        .0.2.jar:
        /usr/lib/hadoop/lib/commons-collections-3
        .2.2.jar:
        /usr/lib/hadoop/lib/jasper-compiler-5
        .5.23.jar:
        /usr/lib/hadoop/lib/jersey-core-1
        .9.jar:
        /usr/lib/hadoop/lib/jackson-mapper-asl-1
        .8.8.jar:
        /usr/lib/hadoop/lib/asm-3
        .2.jar:
        /usr/lib/hadoop/lib/protobuf-java-2
        .5.0.jar:
        /usr/lib/hadoop/lib/curator-framework-2
        .7.1.jar:
        /usr/lib/hadoop/lib/commons-el-1
        .0.jar:
        /usr/lib/hadoop/lib/httpclient-4
        .2.5.jar:
        /usr/lib/hadoop/lib/gson-2
        .2.4.jar:
        /usr/lib/hadoop/lib/commons-compress-1
        .4.1.jar:
        /usr/lib/hadoop/lib/jackson-core-asl-1
        .8.8.jar:
        /usr/lib/hadoop/lib/slf4j-log4j12
        .jar:
        /usr/lib/hadoop/lib/commons-digester-1
        .8.jar:
        /usr/lib/hadoop/lib/commons-lang-2
        .6.jar:
        /usr/lib/hadoop/lib/netty-3
        .6.2.Final.jar:
        /usr/lib/hadoop/lib/paranamer-2
        .3.jar:
        /usr/lib/hadoop/lib/jsr305-3
        .0.0.jar:
        /usr/lib/hadoop/lib/commons-beanutils-1
        .7.0.jar:
        /usr/lib/hadoop/
        .
        //hadoop-common-2
        .6.0-cdh5.7.4-tests.jar:
        /usr/lib/hadoop/
        .
        //hadoop-annotations
        .jar:
        /usr/lib/hadoop/
        .
        //hadoop-annotations-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop/
        .
        //hadoop-nfs
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-generator
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-test-hadoop2
        .jar:
        /usr/lib/hadoop/
        .
        //hadoop-aws-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop/
        .
        //parquet-column
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-pig-bundle
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-scrooge_2
        .10.jar:
        /usr/lib/hadoop/
        .
        //hadoop-auth-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop/
        .
        //hadoop-nfs-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop/
        .
        //hadoop-common-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop/
        .
        //parquet-encoding
        .jar:
        /usr/lib/hadoop/
        .
        //hadoop-aws
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-format-sources
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-tools
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-format-javadoc
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-hadoop
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-jackson
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-common
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-avro
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-thrift
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-format
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-cascading
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-scala_2
        .10.jar:
        /usr/lib/hadoop/
        .
        //hadoop-common
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-pig
        .jar:
        /usr/lib/hadoop/
        .
        //hadoop-auth
        .jar:
        /usr/lib/hadoop/
        .
        //hadoop-common-tests
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-protobuf
        .jar:
        /usr/lib/hadoop/
        .
        //parquet-hadoop-bundle
        .jar:
        /usr/lib/hadoop-hdfs/
        ./:
        /usr/lib/hadoop-hdfs/lib/commons-io-2
        .4.jar:
        /usr/lib/hadoop-hdfs/lib/jasper-runtime-5
        .5.23.jar:
        /usr/lib/hadoop-hdfs/lib/servlet-api-2
        .5.jar:
        /usr/lib/hadoop-hdfs/lib/jersey-server-1
        .9.jar:
        /usr/lib/hadoop-hdfs/lib/xercesImpl-2
        .9.1.jar:
        /usr/lib/hadoop-hdfs/lib/xmlenc-0
        .52.jar:
        /usr/lib/hadoop-hdfs/lib/commons-codec-1
        .4.jar:
        /usr/lib/hadoop-hdfs/lib/jsp-api-2
        .1.jar:
        /usr/lib/hadoop-hdfs/lib/log4j-1
        .2.17.jar:
        /usr/lib/hadoop-hdfs/lib/commons-logging-1
        .1.3.jar:
        /usr/lib/hadoop-hdfs/lib/jetty-util-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop-hdfs/lib/commons-daemon-1
        .0.13.jar:
        /usr/lib/hadoop-hdfs/lib/jetty-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop-hdfs/lib/htrace-core4-4
        .0.1-incubating.jar:
        /usr/lib/hadoop-hdfs/lib/xml-apis-1
        .3.04.jar:
        /usr/lib/hadoop-hdfs/lib/commons-cli-1
        .2.jar:
        /usr/lib/hadoop-hdfs/lib/guava-11
        .0.2.jar:
        /usr/lib/hadoop-hdfs/lib/jersey-core-1
        .9.jar:
        /usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1
        .8.8.jar:
        /usr/lib/hadoop-hdfs/lib/asm-3
        .2.jar:
        /usr/lib/hadoop-hdfs/lib/leveldbjni-all-1
        .8.jar:
        /usr/lib/hadoop-hdfs/lib/protobuf-java-2
        .5.0.jar:
        /usr/lib/hadoop-hdfs/lib/commons-el-1
        .0.jar:
        /usr/lib/hadoop-hdfs/lib/jackson-core-asl-1
        .8.8.jar:
        /usr/lib/hadoop-hdfs/lib/commons-lang-2
        .6.jar:
        /usr/lib/hadoop-hdfs/lib/netty-3
        .6.2.Final.jar:
        /usr/lib/hadoop-hdfs/lib/jsr305-3
        .0.0.jar:
        /usr/lib/hadoop-hdfs/
        .
        //hadoop-hdfs-nfs
        .jar:
        /usr/lib/hadoop-hdfs/
        .
        //hadoop-hdfs-tests
        .jar:
        /usr/lib/hadoop-hdfs/
        .
        //hadoop-hdfs-nfs-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-hdfs/
        .
        //hadoop-hdfs-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-hdfs/
        .
        //hadoop-hdfs
        .jar:
        /usr/lib/hadoop-hdfs/
        .
        //hadoop-hdfs-2
        .6.0-cdh5.7.4-tests.jar:
        /usr/lib/hadoop-yarn/lib/commons-io-2
        .4.jar:
        /usr/lib/hadoop-yarn/lib/guice-servlet-3
        .0.jar:
        /usr/lib/hadoop-yarn/lib/jettison-1
        .1.jar:
        /usr/lib/hadoop-yarn/lib/stax-api-1
        .0-2.jar:
        /usr/lib/hadoop-yarn/lib/zookeeper
        .jar:
        /usr/lib/hadoop-yarn/lib/jline-2
        .11.jar:
        /usr/lib/hadoop-yarn/lib/javax
        .inject-1.jar:
        /usr/lib/hadoop-yarn/lib/servlet-api-2
        .5.jar:
        /usr/lib/hadoop-yarn/lib/aopalliance-1
        .0.jar:
        /usr/lib/hadoop-yarn/lib/jersey-server-1
        .9.jar:
        /usr/lib/hadoop-yarn/lib/activation-1
        .1.jar:
        /usr/lib/hadoop-yarn/lib/guice-3
        .0.jar:
        /usr/lib/hadoop-yarn/lib/commons-codec-1
        .4.jar:
        /usr/lib/hadoop-yarn/lib/jaxb-api-2
        .2.2.jar:
        /usr/lib/hadoop-yarn/lib/jaxb-impl-2
        .2.3-1.jar:
        /usr/lib/hadoop-yarn/lib/log4j-1
        .2.17.jar:
        /usr/lib/hadoop-yarn/lib/commons-logging-1
        .1.3.jar:
        /usr/lib/hadoop-yarn/lib/jetty-util-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop-yarn/lib/jackson-xc-1
        .8.8.jar:
        /usr/lib/hadoop-yarn/lib/jackson-jaxrs-1
        .8.8.jar:
        /usr/lib/hadoop-yarn/lib/jetty-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop-yarn/lib/jersey-guice-1
        .9.jar:
        /usr/lib/hadoop-yarn/lib/commons-httpclient-3
        .1.jar:
        /usr/lib/hadoop-yarn/lib/jersey-json-1
        .9.jar:
        /usr/lib/hadoop-yarn/lib/commons-cli-1
        .2.jar:
        /usr/lib/hadoop-yarn/lib/xz-1
        .0.jar:
        /usr/lib/hadoop-yarn/lib/guava-11
        .0.2.jar:
        /usr/lib/hadoop-yarn/lib/commons-collections-3
        .2.2.jar:
        /usr/lib/hadoop-yarn/lib/jersey-core-1
        .9.jar:
        /usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1
        .8.8.jar:
        /usr/lib/hadoop-yarn/lib/asm-3
        .2.jar:
        /usr/lib/hadoop-yarn/lib/leveldbjni-all-1
        .8.jar:
        /usr/lib/hadoop-yarn/lib/protobuf-java-2
        .5.0.jar:
        /usr/lib/hadoop-yarn/lib/commons-compress-1
        .4.1.jar:
        /usr/lib/hadoop-yarn/lib/jackson-core-asl-1
        .8.8.jar:
        /usr/lib/hadoop-yarn/lib/commons-lang-2
        .6.jar:
        /usr/lib/hadoop-yarn/lib/jersey-client-1
        .9.jar:
        /usr/lib/hadoop-yarn/lib/jsr305-3
        .0.0.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-common-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-registry
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-resourcemanager
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-nodemanager
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-client
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-client-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-tests
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-common
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-nodemanager-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-web-proxy-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-applicationhistoryservice
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-common-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-resourcemanager-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-applications-unmanaged-am-launcher
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-applications-distributedshell
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-applications-distributedshell-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-api
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-tests-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-registry-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-common
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-api-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-applicationhistoryservice-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-server-web-proxy
        .jar:
        /usr/lib/hadoop-yarn/
        .
        //hadoop-yarn-applications-unmanaged-am-launcher-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/lib/commons-io-2
        .4.jar:
        /usr/lib/hadoop-mapreduce/lib/guice-servlet-3
        .0.jar:
        /usr/lib/hadoop-mapreduce/lib/avro
        .jar:
        /usr/lib/hadoop-mapreduce/lib/javax
        .inject-1.jar:
        /usr/lib/hadoop-mapreduce/lib/hamcrest-core-1
        .3.jar:
        /usr/lib/hadoop-mapreduce/lib/aopalliance-1
        .0.jar:
        /usr/lib/hadoop-mapreduce/lib/jersey-server-1
        .9.jar:
        /usr/lib/hadoop-mapreduce/lib/guice-3
        .0.jar:
        /usr/lib/hadoop-mapreduce/lib/log4j-1
        .2.17.jar:
        /usr/lib/hadoop-mapreduce/lib/junit-4
        .11.jar:
        /usr/lib/hadoop-mapreduce/lib/jersey-guice-1
        .9.jar:
        /usr/lib/hadoop-mapreduce/lib/xz-1
        .0.jar:
        /usr/lib/hadoop-mapreduce/lib/snappy-java-1
        .0.4.1.jar:
        /usr/lib/hadoop-mapreduce/lib/jersey-core-1
        .9.jar:
        /usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1
        .8.8.jar:
        /usr/lib/hadoop-mapreduce/lib/asm-3
        .2.jar:
        /usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1
        .8.jar:
        /usr/lib/hadoop-mapreduce/lib/protobuf-java-2
        .5.0.jar:
        /usr/lib/hadoop-mapreduce/lib/commons-compress-1
        .4.1.jar:
        /usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1
        .8.8.jar:
        /usr/lib/hadoop-mapreduce/lib/netty-3
        .6.2.Final.jar:
        /usr/lib/hadoop-mapreduce/lib/paranamer-2
        .3.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-io-2
        .4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jettison-1
        .1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //stax-api-1
        .0-2.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //microsoft-windowsazure-storage-sdk-0
        .6.0.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-rumen-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //zookeeper
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-extras-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-hs-plugins
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //avro
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jsch-0
        .1.42.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //metrics-core-3
        .0.2.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //java-xmlbuilder-0
        .4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //curator-client-2
        .7.1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jasper-runtime-5
        .5.23.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //servlet-api-2
        .5.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-nativetask-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hamcrest-core-1
        .3.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-datajoin
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-net-3
        .1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-openstack
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jets3t-0
        .9.0.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-core-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jackson-databind-2
        .2.3.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //httpcore-4
        .2.5.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-jobclient
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-jobclient-2
        .6.0-cdh5.7.4-tests.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-hs-plugins-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jersey-server-1
        .9.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-gridmix
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //activation-1
        .1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-common
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-sls
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //apacheds-kerberos-codec-2
        .0.0-M15.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //xmlenc-0
        .52.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-beanutils-core-1
        .8.0.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-codec-1
        .4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-datajoin-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jaxb-api-2
        .2.2.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //curator-recipes-2
        .7.1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jsp-api-2
        .1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-openstack-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-common-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //apacheds-i18n-2
        .0.0-M15.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-auth-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jaxb-impl-2
        .2.3-1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //log4j-1
        .2.17.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-logging-1
        .1.3.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-archive-logs-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jetty-util-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jackson-xc-1
        .8.8.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-math3-3
        .1.1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-nativetask
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-archives-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-streaming-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-app
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-azure
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-examples-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-shuffle-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-hs
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jackson-jaxrs-1
        .8.8.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jetty-6
        .1.26.cloudera.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //htrace-core4-4
        .0.1-incubating.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //junit-4
        .11.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //api-util-1
        .0.0-M20.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-jobclient-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-sls-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-httpclient-3
        .1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jersey-json-1
        .9.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //api-asn1-api-1
        .0.0-M20.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-cli-1
        .2.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-jobclient-tests
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-configuration-1
        .6.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //mockito-all-1
        .8.5.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //xz-1
        .0.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-azure-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-app-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //snappy-java-1
        .0.4.1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //guava-11
        .0.2.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-archives
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-rumen
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-collections-3
        .2.2.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jasper-compiler-5
        .5.23.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jersey-core-1
        .9.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jackson-mapper-asl-1
        .8.8.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //asm-3
        .2.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-ant
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jackson-core-2
        .2.3.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-shuffle
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-distcp
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-extras
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //protobuf-java-2
        .5.0.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //curator-framework-2
        .7.1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-hs-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-examples
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-ant-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-el-1
        .0.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //httpclient-4
        .2.5.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //gson-2
        .2.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jackson-annotations-2
        .2.3.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-compress-1
        .4.1.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-auth
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jackson-core-asl-1
        .8.8.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-distcp-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-gridmix-2
        .6.0-cdh5.7.4.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-archive-logs
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-digester-1
        .8.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-lang-2
        .6.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //paranamer-2
        .3.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-streaming
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //jsr305-3
        .0.0.jar:
        /usr/lib/hadoop-mapreduce/
        .
        //hadoop-mapreduce-client-core
        .jar:
        /usr/lib/hadoop-mapreduce/
        .
        //commons-beanutils-1
        .7.0.jar 
       
 
        STARTUP_MSG:   build = http:
        //github
        .com
        /cloudera/hadoop 
        -r 2390c11b3cb7a741189f62797de0d9862f48e211; compiled by 
        'jenkins' 
        on 2016-09-20T23:02Z 
       

        STARTUP_MSG:   java = 1.7.0_75
       

        ************************************************************/
       
 
        2017-11-17 13:09:45,026 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers 
        for 
        [TERM, HUP, INT] 
       

        2017-11-17 13:09:45,030 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
       

        2017-11-17 13:09:45,506 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
       

        2017-11-17 13:09:45,624 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
       

        2017-11-17 13:09:45,624 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
       
 
        2017-11-17 13:09:45,653 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs:
        //bmh 
       
 
        2017-11-17 13:09:45,654 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use bmh to access this namenode
        /service
        . 
       

        2017-11-17 13:09:46,001 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
       
 
        2017-11-17 13:09:46,016 INFO org.apache.hadoop.hdfs.DFSUtil: Starting Web-server 
        for 
        hdfs at: http:
        //name-01
        :50070 
       

        2017-11-17 13:09:46,071 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
       

        2017-11-17 13:09:46,080 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
       
 
        2017-11-17 13:09:46,087 INFO org.apache.hadoop.http.HttpRequestLog: Http request log 
        for 
        http.requests.namenode is not defined 
       
 
        2017-11-17 13:09:46,100 INFO org.apache.hadoop.http.HttpServer2: Added global filter 
        'safety' 
        (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 
       

        2017-11-17 13:09:46,106 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
       

        2017-11-17 13:09:46,106 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
       

        2017-11-17 13:09:46,106 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
       
 
        2017-11-17 13:09:46,142 INFO org.apache.hadoop.http.HttpServer2: Added filter 
        'org.apache.hadoop.hdfs.web.AuthFilter' 
        (class=org.apache.hadoop.hdfs.web.AuthFilter) 
       
 
        2017-11-17 13:09:46,144 INFO org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=
        /webhdfs/v1/
        * 
       

        2017-11-17 13:09:46,162 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 50070
       

        2017-11-17 13:09:46,162 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
       

        2017-11-17 13:09:46,468 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@name-01:50070
       
 
        2017-11-17 13:09:46,508 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.
        dir
        ) configured. Beware of data loss due to lack of redundant storage directories! 
       

        2017-11-17 13:09:46,558 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: No KeyProvider found.
       
 
        2017-11-17 13:09:46,566 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is fair:
        true 
       

        2017-11-17 13:09:46,621 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
       
 
        2017-11-17 13:09:46,622 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-
        hostname
        -check=
        true 
       
 
        2017-11-17 13:09:46,624 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is 
        set 
        to 000:00:00:00.000 
       

        2017-11-17 13:09:46,626 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: The block deletion will start around 2017 Nov 17 13:09:46
       
 
        2017-11-17 13:09:46,628 INFO org.apache.hadoop.util.GSet: Computing capacity 
        for 
        map BlocksMap 
       
 
        2017-11-17 13:09:46,628 INFO org.apache.hadoop.util.GSet: VM 
        type       
        = 64-bit 
       

        2017-11-17 13:09:46,630 INFO org.apache.hadoop.util.GSet: 2.0% max memory 958.5 MB = 19.2 MB
       

        2017-11-17 13:09:46,630 INFO org.apache.hadoop.util.GSet: capacity      = 2^21 = 2097152 entries
       
 
        2017-11-17 13:09:46,641 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.
        enable
        =
        true 
       

        2017-11-17 13:09:46,641 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.key.update.interval=600 min(s), dfs.block.access.token.lifetime=600 min(s), dfs.encrypt.data.transfer.algorithm=null
       

        2017-11-17 13:09:46,797 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication         = 2
       

        2017-11-17 13:09:46,797 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication             = 512
       

        2017-11-17 13:09:46,797 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication             = 1
       

        2017-11-17 13:09:46,797 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams      = 2
       

        2017-11-17 13:09:46,797 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
       
 
        2017-11-17 13:09:46,797 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer        = 
        false 
       

        2017-11-17 13:09:46,797 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
       

        2017-11-17 13:09:46,803 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             = hdfs (auth:SIMPLE)
       

        2017-11-17 13:09:46,803 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          = supergroup
       
 
        2017-11-17 13:09:46,803 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = 
        true 
       

        2017-11-17 13:09:46,803 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Determined nameservice ID: bmh
       
 
        2017-11-17 13:09:46,803 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: 
        true 
       
 
        2017-11-17 13:09:46,805 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: 
        true 
       
 
        2017-11-17 13:09:46,852 INFO org.apache.hadoop.util.GSet: Computing capacity 
        for 
        map INodeMap 
       
 
        2017-11-17 13:09:46,852 INFO org.apache.hadoop.util.GSet: VM 
        type       
        = 64-bit 
       

        2017-11-17 13:09:46,852 INFO org.apache.hadoop.util.GSet: 1.0% max memory 958.5 MB = 9.6 MB
       

        2017-11-17 13:09:46,852 INFO org.apache.hadoop.util.GSet: capacity      = 2^20 = 1048576 entries
       
 
        2017-11-17 13:09:46,854 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching 
        file 
        names occuring 
        more 
        than 10 
        times 
       
 
        2017-11-17 13:09:46,865 INFO org.apache.hadoop.util.GSet: Computing capacity 
        for 
        map cachedBlocks 
       
 
        2017-11-17 13:09:46,865 INFO org.apache.hadoop.util.GSet: VM 
        type       
        = 64-bit 
       

        2017-11-17 13:09:46,865 INFO org.apache.hadoop.util.GSet: 0.25% max memory 958.5 MB = 2.4 MB
       

        2017-11-17 13:09:46,865 INFO org.apache.hadoop.util.GSet: capacity      = 2^18 = 262144 entries
       

        2017-11-17 13:09:46,867 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
       

        2017-11-17 13:09:46,867 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
       

        2017-11-17 13:09:46,867 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
       
 
        2017-11-17 13:09:46,871 INFO org.apache.hadoop.hdfs.server.namenode.
        top
        .metrics.TopMetrics: NNTop conf: dfs.namenode.
        top
        .window.num.buckets = 10 
       
 
        2017-11-17 13:09:46,871 INFO org.apache.hadoop.hdfs.server.namenode.
        top
        .metrics.TopMetrics: NNTop conf: dfs.namenode.
        top
        .num.
        users 
        = 10 
       
 
        2017-11-17 13:09:46,871 INFO org.apache.hadoop.hdfs.server.namenode.
        top
        .metrics.TopMetrics: NNTop conf: dfs.namenode.
        top
        .windows.minutes = 1,5,25 
       

        2017-11-17 13:09:46,872 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
       
 
        2017-11-17 13:09:46,873 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry 
        time 
        is 600000 millis 
       
 
        2017-11-17 13:09:46,875 INFO org.apache.hadoop.util.GSet: Computing capacity 
        for 
        map NameNodeRetryCache 
       
 
        2017-11-17 13:09:46,875 INFO org.apache.hadoop.util.GSet: VM 
        type       
        = 64-bit 
       

        2017-11-17 13:09:46,876 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory 958.5 MB = 294.5 KB
       

        2017-11-17 13:09:46,876 INFO org.apache.hadoop.util.GSet: capacity      = 2^15 = 32768 entries
       
 
        2017-11-17 13:09:46,879 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ACLs enabled? 
        false 
       
 
        2017-11-17 13:09:46,879 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: XAttrs enabled? 
        true 
       

        2017-11-17 13:09:46,879 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Maximum size of an xattr: 16384
       
 
        2017-11-17 13:09:46,887 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on 
        /hdname/in_use
        .lock acquired by nodename 53363@name-01 
       
 
        2017-11-17 13:09:47,773 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Planning to load image: FSImageFile(
        file
        =
        /hdname/current/fsimage_0000000000025720921
        , cpktTxId=0000000000025720921) 
       

        2017-11-17 13:09:49,818 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode: Loading 3109720 INodes.
       
 
        2017-11-17 13:10:01,010 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2984ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =3433ms 
       
 
        GC pool 
        'PS Scavenge' 
        had collection(s): count=1 
        time
        =40ms 
       
 
        2017-11-17 13:10:04,191 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2679ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =3064ms 
       
 
        2017-11-17 13:10:07,361 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2669ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =3044ms 
       
 
        2017-11-17 13:10:09,395 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1534ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1904ms 
       
 
        2017-11-17 13:10:11,453 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1557ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1934ms 
       
 
        2017-11-17 13:10:13,630 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1676ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2060ms 
       
 
        2017-11-17 13:10:15,817 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1686ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2085ms 
       
 
        2017-11-17 13:10:18,025 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1707ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2112ms 
       
 
        2017-11-17 13:10:20,272 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1746ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2156ms 
       
 
        2017-11-17 13:10:22,439 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1666ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2081ms 
       
 
        2017-11-17 13:10:24,630 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1691ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2102ms 
       
 
        2017-11-17 13:10:26,830 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1699ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2115ms 
       
 
        2017-11-17 13:10:29,056 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1725ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2146ms 
       
 
        2017-11-17 13:10:31,385 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1828ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2252ms 
       
 
        2017-11-17 13:10:35,134 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2748ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2830ms 
       
 
        2017-11-17 13:10:37,381 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1246ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1625ms 
       
 
        2017-11-17 13:10:40,385 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2503ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2599ms 
       
 
        2017-11-17 13:10:43,274 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2388ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2389ms 
       
 
        2017-11-17 13:10:46,499 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2224ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2574ms 
       
 
        2017-11-17 13:10:50,074 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2574ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2992ms 
       
 
        2017-11-17 13:10:54,803 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 4228ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =4334ms 
       
 
        2017-11-17 13:10:59,246 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 3942ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =4160ms 
       
 
        2017-11-17 13:11:03,330 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 3583ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =3908ms 
       
 
        2017-11-17 13:11:06,025 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2194ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2598ms 
       
 
        2017-11-17 13:11:08,504 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1978ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2412ms 
       
 
        2017-11-17 13:11:10,137 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1132ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1583ms 
       
 
        2017-11-17 13:11:11,784 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1147ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1606ms 
       
 
        2017-11-17 13:11:13,642 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1357ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1828ms 
       
 
        2017-11-17 13:11:15,375 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1233ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1714ms 
       
 
        2017-11-17 13:11:18,098 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2222ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2708ms 
       
 
        2017-11-17 13:11:19,720 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1121ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1612ms 
       
 
        2017-11-17 13:11:21,282 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1061ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1556ms 
       
 
        2017-11-17 13:11:22,907 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1125ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1621ms 
       
 
        2017-11-17 13:11:24,460 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1052ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1550ms 
       
 
        2017-11-17 13:11:26,013 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1052ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1550ms 
       
 
        2017-11-17 13:11:27,557 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1043ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1542ms 
       
 
        2017-11-17 13:11:29,185 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1127ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1627ms 
       
 
        2017-11-17 13:11:30,955 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1269ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1769ms 
       
 
        2017-11-17 13:11:32,573 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1117ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1617ms 
       
 
        2017-11-17 13:11:35,833 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1077ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =1578ms 
       
 
        2017-11-17 13:12:08,946 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 1252ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=16 
        time
        =26562ms 
       
 
        2017-11-17 13:12:11,521 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2074ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=6 
        time
        =10800ms 
       
 
        2017-11-17 13:12:14,285 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause 
        in 
        JVM or host machine (eg GC): pause of approximately 2264ms 
       
 
        GC pool 
        'PS MarkSweep' 
        had collection(s): count=1 
        time
        =2763ms 
       

        2017-11-17 13:12:14,338 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
       

        java.lang.OutOfMemoryError: GC overhead limit exceeded
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.INodeMap.get(INodeMap.java:92) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getInode(FSDirectory.java:2357) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:207) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:262) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:181) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:946) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:930) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:749) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:680) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:292) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1096) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:778) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) 
       
 
                
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606) 
       

        2017-11-17 13:12:14,343 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
       

        2017-11-17 13:12:14,344 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
       

        /************************************************************
       
 
        SHUTDOWN_MSG: Shutting down NameNode at name-01
        /10
        .0.0.101 
       

        ************************************************************/
       
 
    

   
 

hadoop hdfs jvm内存溢出，无法成功启动

默认启动-Xmx1000m

解决办法：

由于cdh yum安装默认无hadoop-env.sh文件，添加hadoop-env.sh文件，并增加环境参数

 
        export 
        HADOOP_HEAPSIZE=32000

问题解决！

本文转自YU文武貝 51CTO博客，原文链接：http://blog.51cto.com/linuxerxy/1982873，如需转载请自行联系原作者

微信关注我们

原文链接：https://yq.aliyun.com/articles/521099

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

安装PredictionIO

PredictionIO：安装官方地址：http://docs.prediction.io/install/install-linux/ 安装PredictionIO必需安装好java。注意：很多时候安装PredictionIO时会提示：JAVA_HOME 相关的错误，基本上是java版本不对或没有安装。查看是否有安装 java：安装java: 要求是：【Java SE Development Kit 7】即开发工具包，如果已经安装了可以跳过。最好安装 java-jdk-8 及以上稳定性会更高。下载地址：http://www.oracle.com/technetwork/java/javase/downloads/index.html 下载rpm包 rpm -ivh jdk-8u45-linux-x64.rpm 注意：Ubuntu（或基于Debian文件系统）的linux系统不能直接使用rpm 主要是因为Ubuntu的软件包格式是deb，如果要安装rpm包，得把rpm转换成deb，可以使用alien转换工具然后使用dpkg安装，但成功率不高，可以直接下载java的压缩免安...

2017-11-07

626

Hive MapJoin OOM

Hive升级完后ETL开发找到我说有的Job一直failed.看了一下在MAP阶段进行MAPJOIN处理时就OOM了，但是开发说没有加MAPJOIN HINT,其实在0.11后hive.auto.convert.join的默认值变为true也就是会自动去做；并且在0.11加入了一个新的参数hive.ignore.mapjoin.hint来控制是否忽略MAPJOINHINT(HIVE-4042),默认也是true，即使加了默认也会被忽略掉的. 不过对于MAPJOIN对于内存的消耗确实非常高，基本要3X-4X数据量（具体见HIVE-6430），其中HIVE-6418,HIVE-6429都有一定的修复(分别针对MAPJOINKEY,MAPJOINROWCONTAINER)，不过是在0.13了. 本文转自MIKE老毕 51CTO博客，原文链接：http://blog.51cto.com/boylook/1366330，如需转载请自行联系原作者

2017-11-07

780

发表评论

资源下载

更多资源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。