CDH: unable to create new native thread
发现问题 CDH-4.7.1 NameNode is down 启动NameNode报错如下，无法创建新的线程，可能是使用的线程数超过max user processes设定的阈值 2018-08-2608:44:00,532INFOorg.apache.hadoop.http.HttpServer:Jettyboundtoport50070 2018-08-2608:44:00,532INFOorg.mortbay.log:jetty-6.1.26.cloudera.4 2018-08-2608:44:00,773WARNorg.apache.hadoop.security.authentication.server.AuthenticationFilter:'signature.secret'configurationnotset,usingarandomvalueassecret 2018-08-2608:44:00,812INFOorg.mortbay.log:StartedSelectChannelConnector@alish1-dataservice-01.mypna.cn:50070 2018-08-2608:44:00,813INFOorg.apache.hadoop.hdfs.server.namenode.NameNode:Web-serverupat:alish1-dataservice-01.mypna.cn:50070 2018-08-2608:44:00,814INFOorg.apache.hadoop.ipc.Server:IPCServerResponder:starting 2018-08-2608:44:00,815INFOorg.apache.hadoop.ipc.Server:IPCServerlisteneron8020:starting 2018-08-2608:44:00,828INFOorg.apache.hadoop.ipc.Server:IPCServerResponder:starting 2018-08-2608:44:00,828INFOorg.apache.hadoop.ipc.Server:IPCServerlisteneron8022:starting 2018-08-2608:44:00,839FATALorg.apache.hadoop.hdfs.server.namenode.NameNode:Exceptioninnamenodejoin java.lang.OutOfMemoryError:unabletocreatenewnativethread atjava.lang.Thread.start0(NativeMethod) atjava.lang.Thread.start(Thread.java:714) atorg.apache.hadoop.ipc.Server.start(Server.java:2057) atorg.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.start(NameNodeRpcServer.java:303) atorg.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:497) atorg.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:459) atorg.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:621) atorg.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:606) atorg.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1177) atorg.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1241) 2018-08-2608:44:00,851INFOorg.apache.hadoop.util.ExitUtil:Exitingwithstatus1 日志内容如下，检查DNS没有问题，这里没有太多参考意义 #cat/var/log/cloudera-scm-agent/cloudera-scm-agent.log [26/Aug/201807:30:23+0000]4589MainThreadagentINFOPID'19586'associatedwithprocess'1724-hdfs-NAMENODE'withpayload'processname:1724-hdfs-NAMENODEgroupname:1724-hdfs-NAMENODEfrom_state:RUNNINGexpected:0pid:19586'exitedunexpectedly [26/Aug/201807:45:06+0000]4589Monitor-HostMonitorthrottling_loggerERROR(29skipped)Failedtocollectjava-basedDNSnames Traceback(mostrecentcalllast): File"/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py",line53,incollect result,stdout,stderr=self._subprocess_with_timeout(args,self._poll_timeout) File"/usr/lib64/cmf/agent/src/cmf/monitor/host/dns_names.py",line42,in_subprocess_with_timeout returnsubprocess_with_timeout(args,timeout) File"/usr/lib64/cmf/agent/src/cmf/monitor/host/subprocess_timeout.py",line40,insubprocess_with_timeout close_fds=True) File"/usr/lib64/python2.6/subprocess.py",line642,in__init__ errread,errwrite) File"/usr/lib64/python2.6/subprocess.py",line1234,in_execute_child child_exception=pickle.loads(data) OSError:[Errno2]Nosuchfileordirectory 故障排查 这里设置的max user processes为65535已经非常大了，一般来说是达不到这个瓶颈的 #ulimit-a corefilesize(blocks,-c)0 datasegsize(kbytes,-d)unlimited schedulingpriority(-e)0 filesize(blocks,-f)unlimited pendingsignals(-i)127452 maxlockedmemory(kbytes,-l)64 maxmemorysize(kbytes,-m)unlimited openfiles(-n)65535 pipesize(512bytes,-p)8 POSIXmessagequeues(bytes,-q)819200 real-timepriority(-r)0 stacksize(kbytes,-s)10240 cputime(seconds,-t)unlimited maxuserprocesses(-u)65535 virtualmemory(kbytes,-v)unlimited filelocks(-x)unlimited 现在系统的总进程数仅仅一百多个，我们要检查每个进程对应有多少个线程 # ps -ef|wc -l 169 已知这台服务器上主要跑的是java进程，所以重点查看java进程对应的线程数，找到30315这个进程对应约32110个线程，在加上其他进程和线程数，总数超过65535，NameNode无法在申请到多余的线程，所以报错 # pgrep java 1680 5482 19662 28770 30315 35902 # for i in `pgrep java`; do ps -T -p $i |wc -l; done 15 49 30 53 32110 114 # ps -T -p 30315|wc -l 32110 或者通过top -H 命令查看 # top -H top - 10:44:58 up 779 days, 19:34, 3 users, load average: 0.01, 0.05, 0.05 Tasks: 32621 total, 1 running, 32620 sleeping, 0 stopped, 0 zombie Cpu(s): 2.8%us, 4.1%sy, 0.0%ni, 93.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16334284k total, 15879392k used, 454892k free, 381132k buffers Swap: 4194296k total, 0k used, 4194296k free, 8304400k cached 解决方法 找到了问题的原因，我们可以重新设定max user processes的值为100000，再次启动NameNode成功 #echo "100000" > /proc/sys/kernel/threads-max #echo "100000" > /proc/sys/kernel/pid_max （默认32768） #echo "200000" > /proc/sys/vm/max_map_count （默认65530） #vim /etc/security/limits.d/90-nproc.conf * soft nproc unlimited root soft nproc unlimited #vim /etc/security/limits.conf * soft nofile 65535 * hard nofile 65535 * hard nproc 100000 * soft nproc 100000 # ulimit -u 100000
