分布式日志平台--ELKStack实践
在运维系统中，经常遇到如下情况: ①开发人员不能登录线上服务器查看详细日志 ②各个系统都有日志，日志数据分散难以查找 ③日志数据量大，查询速度慢，或者数据不够实时 ④一个调用会涉及多个系统，难以在这些系统的日志中快速定位数据 我们可以采用目前比较流行的ELKStack来满足以上需求。 ELK Stack组成部分: 原理流程图如下: 系统环境: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01~] #cat/etc/redhat-release<br data-filtered="filtered">CentOSrelease6.4(Final)<br data-filtered="filtered">[root@node01~]#uname-a<br data-filtered="filtered">Linuxnode012.6.32-358.el6.x86_64#1SMPFriFeb2200:31:26UTC2013x86_64x86_64x86_64GNU/Linux<br data-filtered="filtered">[root@node01~]#<br data-filtered="filtered"></span> IP地址 运行程序 角色 备注 10.0.0.41 elasticsearch node01 10.0.0.42 elasticsearch node02 10.0.0.43 elasticsearch node03 10.0.0.44 redis kibana kibana 10.0.0.9 nginx logstash web01 10.0.0.10 logstash logstash 实战操作: ①下载安装包安装Java环境以及配置环境变量: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01tools] #ll<br data-filtered="filtered">total289196<br data-filtered="filtered">-rw-r--r--1rootroot28487351Mar2411:29elasticsearch-1.7.5.tar.gz<br data-filtered="filtered">-rw-r--r--1rootroot173271626Mar2411:19jdk-8u45-linux-x64.tar.gz<br data-filtered="filtered">-rw-r--r--1rootroot18560531Mar2411:00kibana-4.1.6-linux-x64.tar.gz<br data-filtered="filtered">-rw-r--r--1rootroot74433726Mar2411:06logstash-2.1.3.tar.gz<br data-filtered="filtered">-rw-r--r--1rootroot1375200Mar2411:03redis-3.0.7.tar.gz<br data-filtered="filtered">[root@node01tools]#<br data-filtered="filtered"></span> 安装jdk(在本次实验中所有服务器上做同样的操作安装jdk,这里以node01为例): 1 <span style= "font-family:'宋体', SimSun;" >[root@node01tools] #tarxfjdk-8u45-linux-x64.tar.gz<br data-filtered="filtered">[root@node01tools]#ll<br data-filtered="filtered">[root@node01tools]#mvjdk1.8.0_45/usr/java/<br data-filtered="filtered">[root@node01tools]#vim/etc/profile<br data-filtered="filtered">exportJAVA_HOME=/usr/java/jdk1.8.0_45<br data-filtered="filtered">exportPATH=$JAVA_HOME/bin:$PATH<br data-filtered="filtered">exportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar<br data-filtered="filtered">[root@node01~]#<br data-filtered="filtered">[root@node01~]#source/etc/profile<br data-filtered="filtered">[root@node01~]#java-version<br data-filtered="filtered">javaversion"1.8.0_45"<br data-filtered="filtered">Java(TM)SERuntimeEnvironment(build1.8.0_45-b14)<br data-filtered="filtered">JavaHotSpot(TM)64-BitServerVM(build25.45-b02,mixedmode)<br data-filtered="filtered">[root@node01~]#<br data-filtered="filtered"></span> ②安装配置elasticsearch（node01、node02、node03上做同样操作） 1 <span style= "font-family:'宋体', SimSun;" >[root@node01tools] #tarxfelasticsearch-1.7.5.tar.gz<br data-filtered="filtered">[root@node01tools]#mvelasticsearch-1.7.5/usr/local/elasticsearch<br data-filtered="filtered">[root@node01tools]#<br data-filtered="filtered">[root@node01tools]#cd/usr/local/elasticsearch/<br data-filtered="filtered">[root@node01elasticsearch]#ll<br data-filtered="filtered">total40<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2411:34bin<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2411:34config<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2411:34lib<br data-filtered="filtered">-rw-rw-r--1rootroot11358Feb217:24LICENSE.txt<br data-filtered="filtered">-rw-rw-r--1rootroot150Feb217:24NOTICE.txt<br data-filtered="filtered">-rw-rw-r--1rootroot8700Feb217:24README.textile<br data-filtered="filtered">[root@node01elasticsearch]#<br data-filtered="filtered">[root@node01elasticsearch]#cdconfig/<br data-filtered="filtered">[root@node01config]#ll<br data-filtered="filtered">total20<br data-filtered="filtered">-rw-rw-r--1rootroot13476Feb217:24elasticsearch.yml<br data-filtered="filtered">-rw-rw-r--1rootroot2054Feb217:24logging.yml<br data-filtered="filtered">[root@node01config]#<br data-filtered="filtered"></span> 修改配置文件如下: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01config] #<br data-filtered="filtered">[root@node01config]#grep-vE'^#|^$'elasticsearch.yml<br data-filtered="filtered">cluster.name:elasticsearch<br data-filtered="filtered">node.name:"node01"<br data-filtered="filtered">node.master:true<br data-filtered="filtered">node.data:true<br data-filtered="filtered">index.number_of_shards:5<br data-filtered="filtered">index.number_of_replicas:1<br data-filtered="filtered">path.conf:/usr/local/elasticsearch/config<br data-filtered="filtered">path.data:/usr/local/elasticsearch/data<br data-filtered="filtered">path.work:/usr/local/elasticsearch/work<br data-filtered="filtered">path.logs:/usr/local/elasticsearch/logs<br data-filtered="filtered">path.plugins:/usr/local/elasticsearch/plugins<br data-filtered="filtered">bootstrap.mlockall:true<br data-filtered="filtered">[root@node01config]#<br data-filtered="filtered"></span> 创建对应的目录: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01elasticsearch] #mkdir-p/usr/local/elasticsearch/{data,work,logs,plugins}<br data-filtered="filtered">[root@node01elasticsearch]#ll<br data-filtered="filtered">total56<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2411:34bin<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2412:52config<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2412:51data<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2411:34lib<br data-filtered="filtered">-rw-rw-r--1rootroot11358Feb217:24LICENSE.txt<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2413:00logs<br data-filtered="filtered">-rw-rw-r--1rootroot150Feb217:24NOTICE.txt<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2413:01plugins<br data-filtered="filtered">-rw-rw-r--1rootroot8700Feb217:24README.textile<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2413:00work<br data-filtered="filtered">[root@node01elasticsearch]#<br data-filtered="filtered"></span> ③启动elasticsearch服务 1 <span style= "font-family:'宋体', SimSun;" >[root@node01elasticsearch] #pwd<br data-filtered="filtered">/usr/local/elasticsearch<br data-filtered="filtered">[root@node01elasticsearch]#ll<br data-filtered="filtered">total44<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2411:34bin<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2412:52config<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2412:51data<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2411:34lib<br data-filtered="filtered">-rw-rw-r--1rootroot11358Feb217:24LICENSE.txt<br data-filtered="filtered">-rw-rw-r--1rootroot150Feb217:24NOTICE.txt<br data-filtered="filtered">-rw-rw-r--1rootroot8700Feb217:24README.textile<br data-filtered="filtered">[root@node01elasticsearch]#/usr/local/elasticsearch/bin/elasticsearch<br data-filtered="filtered">[root@node01elasticsearch]<br data-filtered="filtered"></span> 如果要elasticsearch在后台运行则只需要添加-d，即: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01elasticsearch] #/usr/local/elasticsearch/bin/elasticsearch-d<br data-filtered="filtered"></span> 启动之后查看端口状态: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01elasticsearch] #netstat-lpnut|grepjava<br data-filtered="filtered">tcp00:::9300:::*LISTEN26868/java<br data-filtered="filtered">tcp00:::9200:::*LISTEN26868/java<br data-filtered="filtered">udp00:::54328:::*26868/java<br data-filtered="filtered">[root@node01elasticsearch]#<br data-filtered="filtered"></span> 访问查看状态信息: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01elasticsearch] #curlhttp://10.0.0.41:9200<br data-filtered="filtered">{<br data-filtered="filtered">"status":200,<br data-filtered="filtered">"name":"node01",<br data-filtered="filtered">"cluster_name":"elasticsearch",<br data-filtered="filtered">"version":{<br data-filtered="filtered">"number":"1.7.5",<br data-filtered="filtered">"build_hash":"00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4",<br data-filtered="filtered">"build_timestamp":"2016-02-02T09:55:30Z",<br data-filtered="filtered">"build_snapshot":false,<br data-filtered="filtered">"lucene_version":"4.10.4"<br data-filtered="filtered">},<br data-filtered="filtered">"tagline":"YouKnow,forSearch"<br data-filtered="filtered">}<br data-filtered="filtered">[root@node01elasticsearch]#<br data-filtered="filtered"></span> ④管理elasticsearch服务的脚本(该部分可选) 1 <span style= "font-family:'宋体', SimSun;" >[root@node01elasticsearch] #gitclonehttps://github.com/elastic/elasticsearch-servicewrapper<br data-filtered="filtered">InitializedemptyGitrepositoryin/usr/local/elasticsearch/elasticsearch-servicewrapper/.git/<br data-filtered="filtered">remote:Countingobjects:184,done.<br data-filtered="filtered">remote:Total184(delta0),reused0(delta0),pack-reused184<br data-filtered="filtered">Receivingobjects:100%(184/184),4.55MiB|46KiB/s,done.<br data-filtered="filtered">Resolvingdeltas:100%(53/53),done.<br data-filtered="filtered">[root@node01elasticsearch]#<br data-filtered="filtered">[root@node01elasticsearch]#mvelasticsearch-servicewrapper/service//usr/local/elasticsearch/bin/<br data-filtered="filtered">[root@node01elasticsearch]#/usr/local/elasticsearch/bin/service/elasticsearch<br data-filtered="filtered">Usage:/usr/local/elasticsearch/bin/service/elasticsearch[console|start|stop|restart|condrestart|status|install|remove|dump]<br data-filtered="filtered">Commands:<br data-filtered="filtered">consoleLaunchinthecurrentconsole.<br data-filtered="filtered">startStartinthebackgroundasadaemonprocess.<br data-filtered="filtered">stopStopifrunningasadaemonorinanotherconsole.<br data-filtered="filtered">restartStopifrunningandthenstart.<br data-filtered="filtered">condrestartRestartonlyifalreadyrunning.<br data-filtered="filtered">statusQuerythecurrentstatus.<br data-filtered="filtered">installInstalltostartautomaticallywhensystemboots.<br data-filtered="filtered">removeUninstall.<br data-filtered="filtered">dumpRequestaJavathreaddumpifrunning.<br data-filtered="filtered">[root@node01elasticsearch]#<br data-filtered="filtered"></span> 根据提示，这里我们将其安装到系统启动项中: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01elasticsearch] #/usr/local/elasticsearch/bin/service/elasticsearchinstall<br data-filtered="filtered">DetectedRHELorFedora:<br data-filtered="filtered">InstallingtheElasticsearchdaemon..<br data-filtered="filtered">[root@node01elasticsearch]#chkconfig--list|grepelas<br data-filtered="filtered">elasticsearch0:off1:off2:on3:on4:on5:on6:off<br data-filtered="filtered">[root@node01elasticsearch]#<br data-filtered="filtered"></span> 通过service命令来启动elasticsearch服务 1 <span style= "font-family:'宋体', SimSun;" >[root@node01logs] #serviceelasticsearchstart<br data-filtered="filtered">StartingElasticsearch...<br data-filtered="filtered">WaitingforElasticsearch......<br data-filtered="filtered">running:PID:28084<br data-filtered="filtered">[root@node01logs]#<br data-filtered="filtered"></span> 如果启动报错，即输入如下内容: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01service] #serviceelasticsearchstart<br data-filtered="filtered">StartingElasticsearch...<br data-filtered="filtered">WaitingforElasticsearch......................<br data-filtered="filtered">WARNING:Elasticsearchmayhavefailedtostart.<br data-filtered="filtered"></span> 通过启动日志排查问题: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01logs] #pwd<br data-filtered="filtered">/usr/local/elasticsearch/logs<br data-filtered="filtered">[root@node01logs]#ll<br data-filtered="filtered">total4<br data-filtered="filtered">-rw-r--r--1rootroot3909Mar2413:32service.log<br data-filtered="filtered">[root@node01logs]#catservice.log<br data-filtered="filtered">Theconfiguredwrapper.java.commandcouldnotbefound,attemptingtolaunchanyway:java<br data-filtered="filtered">LaunchingaJVM...<br data-filtered="filtered">VM...<br data-filtered="filtered">jvm3|VM...<br data-filtered="filtered">wrapper|<br data-filtered="filtered">wrapper|------------------------------------------------------------------------<br data-filtered="filtered">wrapper|Advice:<br data-filtered="filtered">wrapper|UsuallywhentheWrapperfailstostarttheJVMprocess,itisbecause<br data-filtered="filtered">wrapper|ofaproblemwiththevalueoftheconfiguredJavacommand.Currently:<br data-filtered="filtered">wrapper|wrapper.java.command=java<br data-filtered="filtered">wrapper|PleasemakesurethatthePATHoranyotherreferencedenvironment<br data-filtered="filtered">wrapper|variablesarecorrectlydefinedforthecurrentenvironment.<br data-filtered="filtered">wrapper|------------------------------------------------------------------------<br data-filtered="filtered">wrapper|<br data-filtered="filtered">wrapper|Theconfiguredwrapper.java.commandcouldnotbefound,attemptingtolaunchanyway:java<br data-filtered="filtered">wrapper|LaunchingaJVM...<br data-filtered="filtered"></span> 根据上述提示，根据实际配置环境将对应参数修改为红色部分，然后重启 1 <span style= "font-family:'宋体', SimSun;" >[root@node01service] #pwd<br data-filtered="filtered">[root@node01service]#<br data-filtered="filtered">[root@node01service]#llelasticsearch.conf<br data-filtered="filtered">-rw-r--r--1rootroot4768Mar2413:32elasticsearch.conf<br data-filtered="filtered">[root@node01service]#<br data-filtered="filtered"></span> 1 <span style= "font-family:'宋体', SimSun;" >[root@node01logs] #serviceelasticsearchstart<br data-filtered="filtered">StartingElasticsearch...<br data-filtered="filtered">WaitingforElasticsearch......<br data-filtered="filtered">running:PID:28084<br data-filtered="filtered">[root@node01logs]#<br data-filtered="filtered"></span> 检查启动状态： 1 <span style= "font-family:'宋体', SimSun;" >[root@node01service] #netstat-lnput<br data-filtered="filtered">ActiveInternetconnections(onlyservers)<br data-filtered="filtered">ProtoRecv-QSend-QLocalAddressForeignAddressStatePID/Programname<br data-filtered="filtered">tcp000.0.0.0:220.0.0.0:*LISTEN968/sshd<br data-filtered="filtered">tcp00127.0.0.1:320000.0.0.0:*LISTEN28086/java<br data-filtered="filtered">tcp00:::9300:::*LISTEN28086/java<br data-filtered="filtered">tcp00:::22:::*LISTEN968/sshd<br data-filtered="filtered">tcp00:::9200:::*LISTEN28086/java<br data-filtered="filtered">udp00:::54328:::*28086/java<br data-filtered="filtered">[root@node01service]#curlhttp://10.0.0.41:9200<br data-filtered="filtered">{<br data-filtered="filtered">"status":200,<br data-filtered="filtered">"name":"node01",<br data-filtered="filtered">"cluster_name":"elasticsearch",<br data-filtered="filtered">"version":{<br data-filtered="filtered">"number":"1.7.5",<br data-filtered="filtered">"build_hash":"00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4",<br data-filtered="filtered">"build_timestamp":"2016-02-02T09:55:30Z",<br data-filtered="filtered">"build_snapshot":false,<br data-filtered="filtered">"lucene_version":"4.10.4"<br data-filtered="filtered">},<br data-filtered="filtered">"tagline":"YouKnow,forSearch"<br data-filtered="filtered">}<br data-filtered="filtered">[root@node01service]#<br data-filtered="filtered"></span> ⑤JAVA API node client Transport client ⑥RESTful API ⑦javascript，.NET,PHP,Perl,Python,Ruby 查询例子: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01service] #curl-XGET'http://10.0.0.41:9200/_count?pretty'-d{"query":{"match_all":{}}}<br data-filtered="filtered">{<br data-filtered="filtered">"count":0,<br data-filtered="filtered">"_shards":{<br data-filtered="filtered">"total":0,<br data-filtered="filtered">"successful":0,<br data-filtered="filtered">"failed":0<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">[root@node01service]#<br data-filtered="filtered"></span> 插件能额外扩展elasticsearch功能，提供各类功能等等。有三种类型的插件： 1. java插件 只包含JAR文件，必须在集群中每个节点上安装而且需要重启才能使插件生效。 2. 网站插件 这类插件包含静态web内容，如js、css、html等等，可以直接从elasticsearch服务，如head插件。只需在一个节点上安装，不需要重启服务。可以通过下面的URL访问，如：http://node-ip:9200/_plugin/plugin_name 3. 混合插件 顾名思义，就是包含上面两种的插件。 安装elasticsearch插件Marvel: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01service] #/usr/local/elasticsearch/bin/plugin-ielasticsearch/marvel/latest<br data-filtered="filtered">->Installingelasticsearch/marvel/latest...<br data-filtered="filtered">Tryinghttp://download.elasticsearch.org/elasticsearch/marvel/marvel-latest.zip...<br data-filtered="filtered">Downloading....................................................................................................................................................................................................................................................................................................................................................DONE<br data-filtered="filtered">Installedelasticsearch/marvel/latestinto/usr/local/elasticsearch/plugins/marvel<br data-filtered="filtered">[root@node01service]#<br data-filtered="filtered"></span> 安装完成之后通过浏览器访问查看集群状况: 1 <span style= "font-family:'宋体', SimSun;" >http: //10 .0.0.41:9200 /_plugin/marvel/kibana/index .html #/dashboard/file/marvel.overview.json<br data-filtered="filtered"></span> 下面是集群节点和索引状况: 选择Dashboards菜单: 点击sense进入如下界面: 编写测试内容: 记录右边生成的id，然后在左边通过id进行查询 如果要进一步查询，则可以做如下操作: ⑧集群管理插件安装 1 <span style= "font-family:'宋体', SimSun;" >[root@node01service] #/usr/local/elasticsearch/bin/plugin-ielasticsearch/marvel/latest<br data-filtered="filtered">->Installingelasticsearch/marvel/latest...<br data-filtered="filtered">Tryinghttp://download.elasticsearch.org/elasticsearch/marvel/marvel-latest.zip...<br data-filtered="filtered">Downloading....................................................................................................................................................................................................................................................................................................................................................DONE<br data-filtered="filtered">Installedelasticsearch/marvel/latestinto/usr/local/elasticsearch/plugins/marvel<br data-filtered="filtered">[root@node01service]#/usr/local/elasticsearch/bin/plugin-imobz/elasticsearch-head<br data-filtered="filtered">->Installingmobz/elasticsearch-head...<br data-filtered="filtered">Tryinghttps://github.com/mobz/elasticsearch-head/archive/master.zip...<br data-filtered="filtered">Downloading...................................................................................................................................................................................................................................................................................DONE<br data-filtered="filtered">Installedmobz/elasticsearch-headinto/usr/local/elasticsearch/plugins/head<br data-filtered="filtered">[root@node01service]#<br data-filtered="filtered"></span> 安装完成后，通过浏览器访问: http://10.0.0.41:9200/_plugin/head/ 部署另外一个节点node02，需要在elasticsearch的配置文件中修改node.name: "node02"即可，其他与node01保持一致 1 <span style= "font-family:'宋体', SimSun;" >[root@node02tools] #cd<br data-filtered="filtered">[root@node02~]#grep'^[a-z]'/usr/local/elasticsearch/config/elasticsearch.yml<br data-filtered="filtered">cluster.name:elasticsearch<br data-filtered="filtered">node.name:"node02"<br data-filtered="filtered">node.master:true<br data-filtered="filtered">node.data:true<br data-filtered="filtered">index.number_of_shards:5<br data-filtered="filtered">index.number_of_replicas:1<br data-filtered="filtered">path.conf:/usr/local/elasticsearch/config<br data-filtered="filtered">path.data:/usr/local/elasticsearch/data<br data-filtered="filtered">path.work:/usr/local/elasticsearch/work<br data-filtered="filtered">path.logs:/usr/local/elasticsearch/logs<br data-filtered="filtered">path.plugins:/usr/local/elasticsearch/plugins<br data-filtered="filtered">bootstrap.mlockall:true<br data-filtered="filtered">[root@node02~]#<br data-filtered="filtered">[root@node02~]#mkdir-p/usr/local/elasticsearch/{data,work,logs.plugins}<br data-filtered="filtered">[root@node02~]#ll/usr/local/elasticsearch/<br data-filtered="filtered">total56<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:23bin<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:23config<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:31data<br data-filtered="filtered">drwxr-xr-x4rootroot4096Mar2414:27elasticsearch-servicewrapper<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2414:23lib<br data-filtered="filtered">-rw-rw-r--1rootroot11358Feb217:24LICENSE.txt<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:31logs.plugins<br data-filtered="filtered">-rw-rw-r--1rootroot150Feb217:24NOTICE.txt<br data-filtered="filtered">-rw-rw-r--1rootroot8700Feb217:24README.textile<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:31work<br data-filtered="filtered">[root@node02~]#<br data-filtered="filtered"></span> 安装管理elasticsearch的管理工具elasticsearch-servicewrapper 1 <span style= "font-family:'宋体', SimSun;" >[root@node02elasticsearch] #gitclonehttps://github.com/elastic/elasticsearch-servicewrapper<br data-filtered="filtered">InitializedemptyGitrepositoryin/usr/local/elasticsearch/elasticsearch-servicewrapper/.git/<br data-filtered="filtered">remote:Countingobjects:184,done.<br data-filtered="filtered">remote:Total184(delta0),reused0(delta0),pack-reused184<br data-filtered="filtered">Receivingobjects:100%(184/184),4.55MiB|10KiB/s,done.<br data-filtered="filtered">Resolvingdeltas:100%(53/53),done.<br data-filtered="filtered">[[root@node02elasticsearch]#ll<br data-filtered="filtered">total80<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2414:33bin<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:23config<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:31data<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2414:33elasticsearch-servicewrapper<br data-filtered="filtered">drwxr-xr-x3rootroot4096Mar2414:23lib<br data-filtered="filtered">-rw-rw-r--1rootroot11358Feb217:24LICENSE.txt<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:31logs<br data-filtered="filtered">-rw-rw-r--1rootroot150Feb217:24NOTICE.txt<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:36plugins<br data-filtered="filtered">-rw-rw-r--1rootroot8700Feb217:24README.textile<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2414:31work<br data-filtered="filtered">-rw-r--r--1rootroot18208Mar2414:35wrapper.log<br data-filtered="filtered">[root@node02elasticsearch]#<br data-filtered="filtered">[root@node02elasticsearch]#mvelasticsearch-servicewrapper/service/usr/local/elasticsearch/bin/<br data-filtered="filtered">[root@node02elasticsearch]#/usr/local/elasticsearch/bin/service/elasticsearch<br data-filtered="filtered">Usage:/usr/local/elasticsearch/bin/service/elasticsearch[console|start|stop|restart|condrestart|status|install|remove|dump]<br data-filtered="filtered">Commands:<br data-filtered="filtered">consoleLaunchinthecurrentconsole.<br data-filtered="filtered">startStartinthebackgroundasadaemonprocess.<br data-filtered="filtered">stopStopifrunningasadaemonorinanotherconsole.<br data-filtered="filtered">restartStopifrunningandthenstart.<br data-filtered="filtered">condrestartRestartonlyifalreadyrunning.<br data-filtered="filtered">statusQuerythecurrentstatus.<br data-filtered="filtered">installInstalltostartautomaticallywhensystemboots.<br data-filtered="filtered">removeUninstall.<br data-filtered="filtered">dumpRequestaJavathreaddumpifrunning.<br data-filtered="filtered">[root@node02elasticsearch]#/usr/local/elasticsearch/bin/service/elasticsearchinstall<br data-filtered="filtered">DetectedRHELorFedora:<br data-filtered="filtered">InstallingtheElasticsearchdaemon..<br data-filtered="filtered">[root@node02elasticsearch]#/etc/init.d/elasticsearchstart<br data-filtered="filtered">StartingElasticsearch...<br data-filtered="filtered">WaitingforElasticsearch..........................<br data-filtered="filtered">running:PID:26753<br data-filtered="filtered">[root@node02elasticsearch]#<br data-filtered="filtered">[root@node02service]#pwd<br data-filtered="filtered">/usr/local/elasticsearch/bin/service<br data-filtered="filtered">[root@node02service]#vimelasticsearch.conf<br data-filtered="filtered"></span> 提示:set.default.ES_HEAP_SIZE值设置小于服务器的物理内存，不能等于实际的内存，否则就会启动失败 接上操作，刷新http://10.0.0.42:9200/_plugin/head/ 查看node01的信息: 概览: 索引: 数据浏览: 基本信息： 符合查询: 以上步骤在上述三个节点node01、node02、node03中分别执行 logstash快速入门 官方建议yum安装: Download and install the public signing key: rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo [logstash-2.2] name=Logstash repository for 2.2.x packages baseurl=http://packages.elastic.co/logstash/2.2/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 And your repository is ready for use. You can install it with: yum install logstash 我这里采用源码解压安装: web01上部署nginx和logstash 1 <span style= "font-family:'宋体', SimSun;" >[root@web01tools] #tarxflogstash-2.1.3.tar.gz<br data-filtered="filtered">[root@web01tools]#mvlogstash-2.1.3/usr/local/logstash<br data-filtered="filtered">[root@web01tools]#cd/usr/local/logstash/<br data-filtered="filtered">[root@web01logstash]#ll<br data-filtered="filtered">total152<br data-filtered="filtered">drwxr-xr-x2rootroot4096Mar2415:50bin<br data-filtered="filtered">-rw-rw-r--1rootroot100805Feb1705:00CHANGELOG.md<br data-filtered="filtered">-rw-rw-r--1rootroot2249Feb1705:00CONTRIBUTORS<br data-filtered="filtered">-rw-rw-r--1rootroot3771Feb1705:05Gemfile<br data-filtered="filtered">-rw-rw-r--1rootroot21970Feb1705:00Gemfile.jruby-1.9.lock<br data-filtered="filtered">drwxr-xr-x4rootroot4096Mar2415:50lib<br data-filtered="filtered">-rw-rw-r--1rootroot589Feb1705:00LICENSE<br data-filtered="filtered">-rw-rw-r--1rootroot149Feb1705:00NOTICE.TXT<br data-filtered="filtered">drwxr-xr-x4rootroot4096Mar2415:50vendor<br data-filtered="filtered">[root@web01logstash]#<br data-filtered="filtered"></span> logstash配置文件格式: 1 <span style= "font-family:'宋体', SimSun;" >input{stdin{}}<br data-filtered= "filtered" >output{<br data-filtered= "filtered" >elasticsearch{hosts=>[ "localhost:9200" ]}<br data-filtered= "filtered" >stdout{codec=>rubydebug}<br data-filtered= "filtered" >}<br data-filtered= "filtered" >< /span > 配置文件结构: # This is a comment. You should use comments to describe # parts of your configuration. 1 <span style= "font-family:'宋体', SimSun;" >input{<br data-filtered= "filtered" >...<br data-filtered= "filtered" >}<br data-filtered= "filtered" >filter{<br data-filtered= "filtered" >...<br data-filtered= "filtered" >}<br data-filtered= "filtered" >output{<br data-filtered= "filtered" >...<br data-filtered= "filtered" >}<br data-filtered= "filtered" >< /span > 插件的结构: 1 <span style= "font-family:'宋体', SimSun;" >input{<br data-filtered= "filtered" > file {<br data-filtered= "filtered" >path=> "/var/log/messages" <br data-filtered= "filtered" > type => "syslog" <br data-filtered= "filtered" >}<br data-filtered= "filtered" > file {<br data-filtered= "filtered" >path=> "/var/log/apache/access.log" <br data-filtered= "filtered" > type => "apache" <br data-filtered= "filtered" >}<br data-filtered= "filtered" >}<br data-filtered= "filtered" >< /span > 插件的数据结构: 数组array Example: path => [ "/var/log/messages", "/var/log/*.log" ] path => "/data/mysql/mysql.log" 布尔类型 Example: ssl_enable => true 字节 Examples: my_bytes => "1113" # 1113 bytes my_bytes => "10MiB" # 10485760 bytes my_bytes => "100kib" # 102400 bytes my_bytes => "180 mb" # 180000000 bytes codeC Example: codec => "json" HASH Example: match => { "field1" => "value1" "field2" => "value2" ... } Number Numbers must be valid numeric values (floating point or integer). Example: port => 33 Password A password is a string with a single value that is not logged or printed. Example: my_password => "password" Pathedit A path is a string that represents a valid operating system path. Example: my_path => "/tmp/logstash" String A string must be a single character sequence. Note that string values are enclosed in quotes, either double or single. Literal quotes in the string need to be escaped with a backslash if they are of the same kind as the string delimiter, i.e. single quotes within a single-quoted string need to be escaped as well as double quotes within a double-quoted string. Example: name => "Hello world" name => 'It\'s a beautiful day' Comments Comments are the same as in perl, ruby, and python. A comment starts with a # character, and does not need to be at the beginning of a line. For example: # this is a comment input { # comments can appear at the end of a line, too # ... } logstash版本不同，插件也会不同，我这里用的logstash-2.1.3.tar.gz https://www.elastic.co/guide/en/logstash/2.1/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts logstash的启动方式为: /usr/local/logstash/bin/logstash -f /usr/local/logstash/logstash.conf & 使用方法如下: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01~] #/usr/local/logstash/bin/logstash-t<br data-filtered="filtered">Error:Usage:<br data-filtered="filtered">/bin/logstashagent[OPTIONS]<br data-filtered="filtered">Options:<br data-filtered="filtered">-f,--configCONFIG_PATHLoadthelogstashconfigfromaspecificfile<br data-filtered="filtered">ordirectory.Ifadirectoryisgiven,all<br data-filtered="filtered">filesinthatdirectorywillbeconcatenated<br data-filtered="filtered">inlexicographicalorderandthenparsedasa<br data-filtered="filtered">singleconfigfile.Youcanalsospecify<br data-filtered="filtered">wildcards(globs)andanymatchedfileswill<br data-filtered="filtered">beloadedintheorderdescribedabove.<br data-filtered="filtered">-eCONFIG_STRINGUsethegivenstringastheconfiguration<br data-filtered="filtered">data.Samesyntaxastheconfigfile.Ifno<br data-filtered="filtered">inputisspecified,thenthefollowingis<br data-filtered="filtered">usedasthedefaultinput:<br data-filtered="filtered">"input{stdin{type=>stdin}}"<br data-filtered="filtered">andifnooutputisspecified,thenthe<br data-filtered="filtered">followingisusedasthedefaultoutput:<br data-filtered="filtered">"output{stdout{codec=>rubydebug}}"<br data-filtered="filtered">Ifyouwishtousebothdefaults,pleaseuse<br data-filtered="filtered">theemptystringforthe'-e'flag.<br data-filtered="filtered">(default:"")<br data-filtered="filtered">-w,--filterworkersCOUNTSetsthenumberoffilterworkerstorun.<br data-filtered="filtered">(default:0)<br data-filtered="filtered">-l,--logFILEWritelogstashinternallogstothegiven<br data-filtered="filtered">file.Withoutthisflag,logstashwillemit<br data-filtered="filtered">logstostandardoutput.<br data-filtered="filtered">-vIncreaseverbosityoflogstashinternallogs.<br data-filtered="filtered">Specifyingoncewillshow'informational'<br data-filtered="filtered">logs.Specifyingtwicewillshow'debug'<br data-filtered="filtered">logs.Thisflagisdeprecated.Youshoulduse<br data-filtered="filtered">--verboseor--debuginstead.<br data-filtered="filtered">--quietQuieterlogstashlogging.Thiscausesonly<br data-filtered="filtered">errorstobeemitted.<br data-filtered="filtered">--verboseMoreverboselogging.Thiscauses'info'<br data-filtered="filtered">levellogstobeemitted.<br data-filtered="filtered">--debugMostverboselogging.Thiscauses'debug'<br data-filtered="filtered">levellogstobeemitted.<br data-filtered="filtered">-V,--versionEmittheversionoflogstashanditsfriends,<br data-filtered="filtered">thenexit.<br data-filtered="filtered">-p,--pluginpathPATHApathofwheretofindplugins.Thisflag<br data-filtered="filtered">canbegivenmultipletimestoinclude<br data-filtered="filtered">multiplepaths.Pluginsareexpectedtobe<br data-filtered="filtered">inaspecificdirectoryhierarchy:<br data-filtered="filtered">'PATH/logstash/TYPE/NAME.rb'whereTYPEis<br data-filtered="filtered">'inputs''filters','outputs'or'codecs'<br data-filtered="filtered">andNAMEisthenameoftheplugin.<br data-filtered="filtered">-t,--configtestCheckconfigurationforvalidsyntaxandthenexit.<br data-filtered="filtered">--[no-]allow-unsafe-shutdownForcelogstashtoexitduringshutdowneven<br data-filtered="filtered">iftherearestillinflighteventsinmemory.<br data-filtered="filtered">Bydefault,logstashwillrefusetoquituntilall<br data-filtered="filtered">receivedeventshavebeenpushedtotheoutputs.<br data-filtered="filtered">(default:false)<br data-filtered="filtered">-h,--helpprinthelp<br data-filtered="filtered"></span> 07-logstash-file-redis-es redis的安装和配置： 1 <span style= "font-family:'宋体', SimSun;" >[root@kibana~] #tarxfredis-3.0.7.tar.gz<br data-filtered="filtered">[root@kibana~]#cdredis-3.0.7<br data-filtered="filtered">[root@kibana~]#makeMALLOC=jemalloc<br data-filtered="filtered">[root@kibana~]#makePREFIX=/application/redis-3.0.7install<br data-filtered="filtered">[root@kibana~]#ln-s/application/redis-3.0.7/application/redis<br data-filtered="filtered">[root@kibana~]#echo"exportPATH=/application/redis/bin:$PATH">>/etc/profile<br data-filtered="filtered">[root@kibana~]#source/etc/profile<br data-filtered="filtered">[root@kibana~]#mkdir-p/application/redis/conf<br data-filtered="filtered">[root@kibana~]#cpredis.conf/application/redis/conf/<br data-filtered="filtered">[root@kibana~]#vim/application/redis/conf/redis.conf<br data-filtered="filtered"></span> 修改bind地址，为本机ip地址 [root@kibana ~]#redis-server /application/redis/conf/redis.conf & 切换到node01，登录node02的redis服务 1 <span style= "font-family:'宋体', SimSun;" >[root@kibana~] #redis-cli-h10.0.0.44-p6379<br data-filtered="filtered">10.0.0.11:6379>info<br data-filtered="filtered">#Server<br data-filtered="filtered">redis_version:3.0.7<br data-filtered="filtered">redis_git_sha1:00000000<br data-filtered="filtered"></span> 修改web01上logstash配置文件如下: 1 <span style= "font-family:'宋体', SimSun;" >[root@web01~] #vim/usr/local/logstash/conf/logstash.conf<br data-filtered="filtered">input{<br data-filtered="filtered">file{<br data-filtered="filtered">#type=>"accesslog"<br data-filtered="filtered">path=>"/application/nginx/logs/access.log"<br data-filtered="filtered">start_position=>"beginning"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">output{<br data-filtered="filtered">redis{<br data-filtered="filtered">data_type=>"list"<br data-filtered="filtered">key=>"system-messages"<br data-filtered="filtered">host=>"10.0.0.44"<br data-filtered="filtered">port=>"6379"<br data-filtered="filtered">db=>"0"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">[root@web01~]#<br data-filtered="filtered"></span> 这里用nginx访问日志作为测试，定义输入文件为"/application/nginx/logs/access.log ",输出文件存储到redis里面，定义key为 "system-messages",host为10.0.0.44,端口6379，存储数据库库为db=0。 保证nginx是运行状态: 1 <span style= "font-family:'宋体', SimSun;" >[root@web01~] #ps-ef|grepnginx<br data-filtered="filtered">root11091012:59?00:00:00nginx:masterprocess/application/nginx/sbin/nginx<br data-filtered="filtered">nobody11131109012:59?00:00:00nginx:workerprocess<br data-filtered="filtered">nobody11141109012:59?00:00:00nginx:workerprocess<br data-filtered="filtered">nobody11151109012:59?00:00:00nginx:workerprocess<br data-filtered="filtered">nobody11161109012:59?00:00:00nginx:workerprocess<br data-filtered="filtered">root15721185014:06pts/000:00:00grepnginx<br data-filtered="filtered">[root@web01~]#<br data-filtered="filtered">[root@web01~]#tail-f/application/nginx/logs/access.log<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">10.0.0.10--[17/Aug/2016:14:10:02+0800]"GET/index.htmlHTTP/1.0"200612"-""ApacheBench/2.3"<br data-filtered="filtered">[root@web01~]#<br data-filtered="filtered"></span> 登录redis，查看数据是否已经传入到redis上: 1 <span style= "font-family:'宋体', SimSun;" >[root@kibana~] #redis-cli<br data-filtered="filtered">127.0.0.1:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">127.0.0.1:6379><br data-filtered="filtered">127.0.0.1:6379><br data-filtered="filtered">127.0.0.1:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">127.0.0.1:6379><br data-filtered="filtered"></span> 发现通过浏览器访问的日志已经存入到了redis中，这一步实现日志的转存，下一步实现日志的从redis中通过logstash转存到elasticsearch中 以上实现了数据从数据源（access.log）中转存到redis中，下一步实现从redis中转存到elasticsearch中 这里在logstash上开启一个logstash，并且编写配置文件/usr/local/logstash/logstash.conf 1 <span style= "font-family:'宋体', SimSun;" >[root@logstash~] #cd/usr/local/logstash<br data-filtered="filtered">[root@logstashlogstash]#pwd<br data-filtered="filtered">/usr/local/logstash<br data-filtered="filtered">[root@logstashlogstash]#<br data-filtered="filtered">[root@node02conf]#ps-ef|greplogstash<br data-filtered="filtered">root4307242030015:46pts/100:00:00greplogstash<br data-filtered="filtered">[root@node02conf]#cat/usr/local/logstash/logstash.conf<br data-filtered="filtered">input{<br data-filtered="filtered">redis{<br data-filtered="filtered">data_type=>"list"<br data-filtered="filtered">key=>"system-messages"<br data-filtered="filtered">host=>"10.0.0.44"<br data-filtered="filtered">port=>"6379"<br data-filtered="filtered">db=>"0"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">output{<br data-filtered="filtered">elasticsearch{<br data-filtered="filtered">hosts=>"10.0.0.41"<br data-filtered="filtered">#protocol=>"http"<br data-filtered="filtered">index=>"redis-messages-%{+YYYY.MM.dd}"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">[root@logstashconf]#<br data-filtered="filtered"></span> 说明:定义数据写入文件为redis,对应的键和主机以及端口如下: 1 <span style= "font-family:'宋体', SimSun;" >redis{<br data-filtered= "filtered" >data_type=> "list" <br data-filtered= "filtered" >key=> "system-messages" <br data-filtered= "filtered" >host=> "10.0.0.44" <br data-filtered= "filtered" >port=> "6379" <br data-filtered= "filtered" >db=> "0" <br data-filtered= "filtered" >}<br data-filtered= "filtered" >< /span > 数据输出到elasticsearch中，具体配置为: 1 <span style= "font-family:'宋体', SimSun;" >elasticsearch{<br data-filtered= "filtered" >hosts=> "10.0.0.41" <br data-filtered= "filtered" >index=> "system-redis-messages-%{+YYYY.MM.dd}" <br data-filtered= "filtered" >}<br data-filtered= "filtered" >< /span > 启动logstash进行日志收集 1 <span style= "font-family:'宋体', SimSun;" >[root@logstashconf] #/usr/local/logstash/bin/logstash-f/usr/local/logstash/conf/logstash.conf&<br data-filtered="filtered">[1]43097<br data-filtered="filtered">[root@node02conf]#<br data-filtered="filtered">[root@node02conf]#<br data-filtered="filtered">[root@node02conf]#<br data-filtered="filtered">[root@node02conf]#<br data-filtered="filtered">[root@node02conf]#<br data-filtered="filtered">[root@node02conf]#<br data-filtered="filtered">[root@node02conf]#ps-ef|greplogstash<br data-filtered="filtered">[root@logstashlogstash]#ps-ef|greplogstash<br data-filtered="filtered">root11691141713:11pts/000:04:30/usr/java/jdk1.8.0_73/bin/java-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-XX:+HeapDumpOnOutOfMemoryError-Xmx1g-Xss2048k-Djffi.boot.library.path=/usr/local/logstash/vendor/jruby/lib/jni-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=/usr/local/logstash/heapdump.hprof-Xbootclasspath/a:/usr/local/logstash/vendor/jruby/lib/jruby.jar-classpath:-Djruby.home=/usr/local/logstash/vendor/jruby-Djruby.lib=/usr/local/logstash/vendor/jruby/lib-Djruby.script=jruby-Djruby.shell=/bin/shorg.jruby.Main--1.9/usr/local/logstash/lib/bootstrap/environment.rblogstash/runner.rbagent-f/usr/local/logstash/conf/logstash.conf<br data-filtered="filtered">root13391141014:14pts/000:00:00greplogstash<br data-filtered="filtered">[root@logstashlogstash]#<br data-filtered="filtered"></span> 再回过来查看redis的db中的数据是否已经转存过去（即存储到elasticserach中） 1 <span style= "font-family:'宋体', SimSun;" >127.0.0.1:6379>LLENsystem-messages<br data-filtered= "filtered" >(integer)0<br data-filtered= "filtered" >127.0.0.1:6379><br data-filtered= "filtered" >127.0.0.1:6379><br data-filtered= "filtered" >127.0.0.1:6379>keys*<br data-filtered= "filtered" >(emptylistor set )<br data-filtered= "filtered" >127.0.0.1:6379><br data-filtered= "filtered" >< /span > 从上述输入可以知道数据应转存到elasticsearch中，下面我可以进行查看: 进一步查看日志信息: logstash收集json格式的nginx日志，然后将日志转存到elasticsearch中 json格式分割nginx日志配置参数: log_format logstash_json '{ "@timestamp": "$time_local", ' '"@fields": { ' '"remote_addr": "$remote_addr", ' '"remote_user": "$remote_user", ' '"body_bytes_sent": "$body_bytes_sent", ' '"request_time": "$request_time", ' '"status": "$status", ' '"request": "$request", ' '"request_method": "$request_method", ' '"http_referrer": "$http_referer", ' '"body_bytes_sent":"$body_bytes_sent", ' '"http_x_forwarded_for": "$http_x_forwarded_for", ' '"http_user_agent": "$http_user_agent" } }'; 修改后的nginx配置文件为: 1 <span style= "font-family:'宋体', SimSun;" >[root@web01conf] #pwd<br data-filtered="filtered">/application/nginx/conf<br data-filtered="filtered">[root@web01conf]#catnginx.conf<br data-filtered="filtered">worker_processes4;<br data-filtered="filtered">events{<br data-filtered="filtered">worker_connections1024;<br data-filtered="filtered">}<br data-filtered="filtered">http{<br data-filtered="filtered">includemime.types;<br data-filtered="filtered">default_typeapplication/octet-stream;<br data-filtered="filtered">log_formatlogstash_json'{"@timestamp":"$time_local",'<br data-filtered="filtered">'"@fields":{'<br data-filtered="filtered">'"remote_addr":"$remote_addr",'<br data-filtered="filtered">'"remote_user":"$remote_user",'<br data-filtered="filtered">'"body_bytes_sent":"$body_bytes_sent",'<br data-filtered="filtered">'"request_time":"$request_time",'<br data-filtered="filtered">'"status":"$status",'<br data-filtered="filtered">'"request":"$request",'<br data-filtered="filtered">'"request_method":"$request_method",'<br data-filtered="filtered">'"http_referrer":"$http_referer",'<br data-filtered="filtered">'"body_bytes_sent":"$body_bytes_sent",'<br data-filtered="filtered">'"http_x_forwarded_for":"$http_x_forwarded_for",'<br data-filtered="filtered">'"http_user_agent":"$http_user_agent"}}';<br data-filtered="filtered">sendfileon;<br data-filtered="filtered">keepalive_timeout65;<br data-filtered="filtered">server{<br data-filtered="filtered">listen80;<br data-filtered="filtered">server_namelocalhost;<br data-filtered="filtered">access_loglogs/access_json.loglogstash_json;<br data-filtered="filtered">location/{<br data-filtered="filtered">roothtml;<br data-filtered="filtered">indexindex.htmlindex.htm;<br data-filtered="filtered">}<br data-filtered="filtered">error_page500502503504/50x.html;<br data-filtered="filtered">location=/50x.html{<br data-filtered="filtered">roothtml;<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">[root@node01conf]#<br data-filtered="filtered"></span> 启动nginx： 1 <span style= "font-family:'宋体', SimSun;" >[root@web01nginx] #killallnginx<br data-filtered="filtered">nginx:noprocesskilled<br data-filtered="filtered">[root@web01nginx]#<br data-filtered="filtered">[root@web01nginx]#/application/nginx/sbin/nginx-t<br data-filtered="filtered">nginx:theconfigurationfile/application/nginx-1.6.0/conf/nginx.confsyntaxisok<br data-filtered="filtered">nginx:configurationfile/application/nginx-1.6.0/conf/nginx.conftestissuccessful<br data-filtered="filtered">[root@web01nginx]#<br data-filtered="filtered">[root@web01nginx]#cd-<br data-filtered="filtered">/application/nginx/logs<br data-filtered="filtered">[root@web01logs]#ll<br data-filtered="filtered">total9976<br data-filtered="filtered">-rw-r--r--1rootroot0Aug1714:27access_json.log<br data-filtered="filtered">-rw-r--r--.1rootroot10056370Aug1714:26access.log<br data-filtered="filtered">-rw-r--r--.1rootroot146445Aug1700:15error.log<br data-filtered="filtered">-rw-r--r--1rootroot5Aug1714:27nginx.pid<br data-filtered="filtered">[root@web01logs]#<br data-filtered="filtered"></span> 通过ab命令访问本地nginx服务: 1 <span style= "font-family:'宋体', SimSun;" >[root@logstashconf] #ab-n10000-c50http://10.0.0.9/index.html<br data-filtered="filtered">ThisisApacheBench,Version2.3<$Revision:655654$><br data-filtered="filtered">Copyright1996AdamTwiss,ZeusTechnologyLtd,http://www.zeustech.net/<br data-filtered="filtered">LicensedtoTheApacheSoftwareFoundation,http://www.apache.org/<br data-filtered="filtered">Benchmarking10.0.0.9(bepatient)<br data-filtered="filtered">Completed1000requests<br data-filtered="filtered">Completed2000requests<br data-filtered="filtered">Completed3000requests<br data-filtered="filtered">Completed4000requests<br data-filtered="filtered">Completed5000requests<br data-filtered="filtered">Completed6000requests<br data-filtered="filtered">Completed7000requests<br data-filtered="filtered">Completed8000requests<br data-filtered="filtered">Completed9000requests<br data-filtered="filtered">Completed10000requests<br data-filtered="filtered">Finished10000requests<br data-filtered="filtered">ServerSoftware:nginx/1.6.2<br data-filtered="filtered">ServerHostname:10.0.0.9<br data-filtered="filtered">ServerPort:80<br data-filtered="filtered">DocumentPath:/index.html<br data-filtered="filtered">DocumentLength:612bytes<br data-filtered="filtered">ConcurrencyLevel:50<br data-filtered="filtered">Timetakenfortests:2.277seconds<br data-filtered="filtered">Completerequests:10000<br data-filtered="filtered">Failedrequests:0<br data-filtered="filtered">Writeerrors:0<br data-filtered="filtered">Totaltransferred:8447596bytes<br data-filtered="filtered">HTMLtransferred:6125508bytes<br data-filtered="filtered">Requestspersecond:4391.57[#/sec](mean)<br data-filtered="filtered">Timeperrequest:11.385[ms](mean)<br data-filtered="filtered">Timeperrequest:0.228[ms](mean,acrossallconcurrentrequests)<br data-filtered="filtered">Transferrate:3622.87[Kbytes/sec]received<br data-filtered="filtered">ConnectionTimes(ms)<br data-filtered="filtered">minmean[+/-sd]medianmax<br data-filtered="filtered">Connect:022.5227<br data-filtered="filtered">Processing:294.9950<br data-filtered="filtered">Waiting:185.1849<br data-filtered="filtered">Total:3114.81154<br data-filtered="filtered">Percentageoftherequestsservedwithinacertaintime(ms)<br data-filtered="filtered">50%11<br data-filtered="filtered">66%11<br data-filtered="filtered">75%12<br data-filtered="filtered">80%13<br data-filtered="filtered">90%15<br data-filtered="filtered">95%19<br data-filtered="filtered">98%26<br data-filtered="filtered">99%33<br data-filtered="filtered">100%54(longestrequest)<br data-filtered="filtered">[root@logstashconf]#<br data-filtered="filtered"></span> 访问nginx产生的日志为: 1 <span style= "font-family:'宋体', SimSun;" >[root@web01logs] #head-n10access_json.log<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.006","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.000","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.000","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.007","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.007","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.007","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.007","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.007","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.007","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">{"@timestamp":"17/Aug/2016:14:27:49+0800","@fields":{"remote_addr":"10.0.0.10","remote_user":"-","body_bytes_sent":"612","request_time":"0.007","status":"200","request":"GET/index.htmlHTTP/1.0","request_method":"GET","http_referrer":"-","body_bytes_sent":"612","http_x_forwarded_for":"-","http_user_agent":"ApacheBench/2.3"}}<br data-filtered="filtered">[root@web01logs]#<br data-filtered="filtered"></span> logstash的配置文件文件为: 1 <span style= "font-family:'宋体', SimSun;" >[root@node01logs] #vim/usr/local/logstash/conf/logstash.conf<br data-filtered="filtered">input{<br data-filtered="filtered">file{<br data-filtered="filtered">#type=>"accesslog"<br data-filtered="filtered">path=>"/application/nginx/logs/access_json.log"<br data-filtered="filtered">start_position=>"beginning"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">output{<br data-filtered="filtered">redis{<br data-filtered="filtered">data_type=>"list"<br data-filtered="filtered">key=>"system-messages"<br data-filtered="filtered">host=>"10.0.0.44"<br data-filtered="filtered">port=>"6379"<br data-filtered="filtered">db=>"0"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">[root@node01logs]#<br data-filtered="filtered"></span> 重启logstash 1 <span style= "font-family:'宋体', SimSun;" >[root@web01logs] #/usr/local/logstash/bin/logstash-f/usr/local/logstash/conf/logstash.conf&<br data-filtered="filtered">[1]20328<br data-filtered="filtered">[root@web01logs]#<br data-filtered="filtered">[root@web01logs]#<br data-filtered="filtered">[root@web01logs]#<br data-filtered="filtered">[root@web01logs]#<br data-filtered="filtered">[root@web01logs]#ps-ef|greplogstash<br data-filtered="filtered">root163811859914:32pts/000:03:56/usr/java/jdk1.8.0_73/bin/java-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-XX:+HeapDumpOnOutOfMemoryError-Xmx1g-Xss2048k-Djffi.boot.library.path=/usr/local/logstash/vendor/jruby/lib/jni-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=/usr/local/logstash/heapdump.hprof-Xbootclasspath/a:/usr/local/logstash/vendor/jruby/lib/jruby.jar-classpath:-Djruby.home=/usr/local/logstash/vendor/jruby-Djruby.lib=/usr/local/logstash/vendor/jruby/lib-Djruby.script=jruby-Djruby.shell=/bin/shorg.jruby.Main--1.9/usr/local/logstash/lib/bootstrap/environment.rblogstash/runner.rbagent-f/usr/local/logstash/conf/logstash.conf<br data-filtered="filtered">root16781185014:34pts/000:00:00greplogstash<br data-filtered="filtered">[root@web01~]#<br data-filtered="filtered"></span> 连接redis查看日志是否写入： 1 <span style= "font-family:'宋体', SimSun;" >[root@kibana] #redis-cli-h10.0.0.44-p6379<br data-filtered="filtered">10.0.0.44:6379>select2<br data-filtered="filtered">OK<br data-filtered="filtered">10.0.0.44:6379[2]>keys*<br data-filtered="filtered">(emptylistorset)<br data-filtered="filtered">10.0.0.44:6379[2]><br data-filtered="filtered"></span> 通过ab命令访问nginx， 1 <span style= "font-family:'宋体', SimSun;" >[root@node01~] #ab-n1000-c100http://10.0.0.9/<br data-filtered="filtered">ThisisApacheBench,Version2.3<$Revision:655654$><br data-filtered="filtered">Copyright1996AdamTwiss,ZeusTechnologyLtd,http://www.zeustech.net/<br data-filtered="filtered">LicensedtoTheApacheSoftwareFoundation,http://www.apache.org/<br data-filtered="filtered">Benchmarking10.0.0.10(bepatient)<br data-filtered="filtered">Completed100requests<br data-filtered="filtered">Completed200requests<br data-filtered="filtered">Completed300requests<br data-filtered="filtered">Completed400requests<br data-filtered="filtered">Completed500requests<br data-filtered="filtered">Completed600requests<br data-filtered="filtered">Completed700requests<br data-filtered="filtered">Completed800requests<br data-filtered="filtered">Completed900requests<br data-filtered="filtered">Completed1000requests<br data-filtered="filtered">Finished1000requests<br data-filtered="filtered">ServerSoftware:nginx/1.6.2<br data-filtered="filtered">ServerHostname:10.0.0.9<br data-filtered="filtered">ServerPort:80<br data-filtered="filtered">DocumentPath:/<br data-filtered="filtered">DocumentLength:612bytes<br data-filtered="filtered">ConcurrencyLevel:100<br data-filtered="filtered">Timetakenfortests:0.274seconds<br data-filtered="filtered">Completerequests:1000<br data-filtered="filtered">Failedrequests:0<br data-filtered="filtered">Writeerrors:0<br data-filtered="filtered">Totaltransferred:881136bytes<br data-filtered="filtered">HTMLtransferred:638928bytes<br data-filtered="filtered">Requestspersecond:3649.85[#/sec](mean)<br data-filtered="filtered">Timeperrequest:27.398[ms](mean)<br data-filtered="filtered">Timeperrequest:0.274[ms](mean,acrossallconcurrentrequests)<br data-filtered="filtered">Transferrate:3140.64[Kbytes/sec]received<br data-filtered="filtered">ConnectionTimes(ms)<br data-filtered="filtered">minmean[+/-sd]medianmax<br data-filtered="filtered">Connect:084.3915<br data-filtered="filtered">Processing:4188.61651<br data-filtered="filtered">Waiting:1159.31249<br data-filtered="filtered">Total:12265.72552<br data-filtered="filtered">Percentageoftherequestsservedwithinacertaintime(ms)<br data-filtered="filtered">50%25<br data-filtered="filtered">66%27<br data-filtered="filtered">75%28<br data-filtered="filtered">80%29<br data-filtered="filtered">90%34<br data-filtered="filtered">95%38<br data-filtered="filtered">98%42<br data-filtered="filtered">99%42<br data-filtered="filtered">100%52(longestrequest)<br data-filtered="filtered">[root@node01~]#<br data-filtered="filtered"></span> 再次检查redis是否写入数据: 1 <span style= "font-family:'宋体', SimSun;" >[root@kibana] #redis-cli-h10.0.0.44-p6379<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">(emptylistorset)<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">10.0.0.44:6379><br data-filtered="filtered"></span> 数据成功写入redis，接下来将redis数据写入elasticsearch中 在logstash主机上编写logstash配置文件 1 <span style= "font-family:'宋体', SimSun;" >[root@logstash] #cd/usr/local/logstash/conf<br data-filtered="filtered">[root@logstashconf]#catlogstash.conf<br data-filtered="filtered">input{<br data-filtered="filtered">redis{<br data-filtered="filtered">data_type=>"list"<br data-filtered="filtered">key=>"system-messages"<br data-filtered="filtered">host=>"10.0.0.44"<br data-filtered="filtered">port=>"6379"<br data-filtered="filtered">db=>"0"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">output{<br data-filtered="filtered">elasticsearch{<br data-filtered="filtered">hosts=>"10.0.0.41"<br data-filtered="filtered">index=>"nginx-access-log-%{+YYYY.MM.dd}"<br data-filtered="filtered">}<br data-filtered="filtered">}<br data-filtered="filtered">[root@logstashconf]#<br data-filtered="filtered"></span> 测试文件是否正确，然后重启logstash 1 <span style= "font-family:'宋体', SimSun;" >[root@logstashconf] #/usr/local/logstash/bin/logstash-t-f/usr/local/logstash/conf/logstash.conf<br data-filtered="filtered">ConfigurationOK<br data-filtered="filtered">[root@logstashconf]#<br data-filtered="filtered">[root@logstashconf]#/usr/local/logstash/bin/logstash-f/usr/local/logstash/conf/logstash.conf&<br data-filtered="filtered">[1]44494<br data-filtered="filtered">[root@logstashconf]#<br data-filtered="filtered"></span> 用ab进行测试访问: 1 <span style= "font-family:'宋体', SimSun;" >[root@logstash~] #ab-n10000-c100http://10.0.0.41/<br data-filtered="filtered">ThisisApacheBench,Version2.3<$Revision:655654$><br data-filtered="filtered">Copyright1996AdamTwiss,ZeusTechnologyLtd,http://www.zeustech.net/<br data-filtered="filtered">LicensedtoTheApacheSoftwareFoundation,http://www.apache.org/<br data-filtered="filtered">Benchmarking10.0.0.10(bepatient)<br data-filtered="filtered">Completed1000requests<br data-filtered="filtered">Completed2000requests<br data-filtered="filtered">Completed3000requests<br data-filtered="filtered">Completed4000requests<br data-filtered="filtered">Completed5000requests<br data-filtered="filtered">Completed6000requests<br data-filtered="filtered">Completed7000requests<br data-filtered="filtered">Completed8000requests<br data-filtered="filtered">Completed9000requests<br data-filtered="filtered">Completed10000requests<br data-filtered="filtered">Finished10000requests<br data-filtered="filtered">ServerSoftware:nginx/1.6.2<br data-filtered="filtered">ServerHostname:10.0.0.10<br data-filtered="filtered">ServerPort:80<br data-filtered="filtered">DocumentPath:/<br data-filtered="filtered">DocumentLength:612bytes<br data-filtered="filtered">ConcurrencyLevel:100<br data-filtered="filtered">Timetakenfortests:2.117seconds<br data-filtered="filtered">Completerequests:10000<br data-filtered="filtered">Failedrequests:0<br data-filtered="filtered">Writeerrors:0<br data-filtered="filtered">Totaltransferred:8510052bytes<br data-filtered="filtered">HTMLtransferred:6170796bytes<br data-filtered="filtered">Requestspersecond:4722.57[#/sec](mean)<br data-filtered="filtered">Timeperrequest:21.175[ms](mean)<br data-filtered="filtered">Timeperrequest:0.212[ms](mean,acrossallconcurrentrequests)<br data-filtered="filtered">Transferrate:3924.74[Kbytes/sec]received<br data-filtered="filtered">ConnectionTimes(ms)<br data-filtered="filtered">minmean[+/-sd]medianmax<br data-filtered="filtered">Connect:043.7325<br data-filtered="filtered">Processing:21710.81575<br data-filtered="filtered">Waiting:11510.31275<br data-filtered="filtered">Total:52110.91875<br data-filtered="filtered">Percentageoftherequestsservedwithinacertaintime(ms)<br data-filtered="filtered">50%18<br data-filtered="filtered">66%24<br data-filtered="filtered">75%27<br data-filtered="filtered">80%29<br data-filtered="filtered">90%36<br data-filtered="filtered">95%44<br data-filtered="filtered">98%49<br data-filtered="filtered">99%57<br data-filtered="filtered">100%75(longestrequest)<br data-filtered="filtered">[root@node01~]#<br data-filtered="filtered"></span> redis查看数据写入 1 <span style= "font-family:'宋体', SimSun;" >[root@kibana~] #redis-cli-h10.0.0.44-p6379<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">(emptylistorset)<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">1)"system-messages"<br data-filtered="filtered">10.0.0.44:6379>keys*<br data-filtered="filtered">(emptylistorset)<br data-filtered="filtered">10.0.0.44:6379><br data-filtered="filtered"></span> 查看elasticsearch数据 进一步查看数据 数据格式为键值形式，即json格式 运用kibana插件对elasticsearch+logstash+redis收集到的数据进行可视化展示 解压: 1 <span style= "font-family:'宋体', SimSun;" >[root@kibana~] #cd/opt/tools<br data-filtered="filtered">[root@kibanatools]#tarxfkibana-4.1.6-linux-x64.tar.gz<br data-filtered="filtered">[root@kibanatools]#mvkibana-4.1.6-linux-x64/usr/local/kibana<br data-filtered="filtered">[root@kibanatools]#cd/usr/local/kibana/<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered"></span> 配置: 1 <span style= "font-family:'宋体', SimSun;" >[root@kibanaconfig] #pwd<br data-filtered="filtered">/usr/local/kibana/config<br data-filtered="filtered">[root@kibanaconfig]#ll<br data-filtered="filtered">total4<br data-filtered="filtered">-rw-r--r--1logstashgames2933Mar1003:29kibana.yml<br data-filtered="filtered">[root@kibanaconfig]#<br data-filtered="filtered">[root@kibanaconfig]#vimkibana.yml<br data-filtered="filtered"></span> 修改为: 1 <span style= "font-family:'宋体', SimSun;" >[root@kibanaconfig] #cd..<br data-filtered="filtered">[root@kibanakibana]#pwd<br data-filtered="filtered">/usr/local/kibana<br data-filtered="filtered">[root@kibanakibana]#ll<br data-filtered="filtered">total28<br data-filtered="filtered">drwxr-xr-x2logstashgames4096Mar2823:24bin<br data-filtered="filtered">drwxr-xr-x2logstashgames4096Mar2823:31config<br data-filtered="filtered">-rw-r--r--1logstashgames563Mar1003:29LICENSE.txt<br data-filtered="filtered">drwxr-xr-x6logstashgames4096Mar2823:24node<br data-filtered="filtered">drwxr-xr-x2logstashgames4096Mar2823:24plugins<br data-filtered="filtered">-rw-r--r--1logstashgames2510Mar1003:29README.txt<br data-filtered="filtered">drwxr-xr-x9logstashgames4096Mar1003:29src<br data-filtered="filtered">[root@kibanakibana]#./bin/kibana-h<br data-filtered="filtered">Usage:kibana[options]<br data-filtered="filtered">Kibanaisanopensource(ApacheLicensed),browserbasedanalyticsandsearchdashboardforElasticsearch.<br data-filtered="filtered">Options:<br data-filtered="filtered">-h,--helpoutputusageinformation<br data-filtered="filtered">-V,--versionoutputtheversionnumber<br data-filtered="filtered">-e,--elasticsearch<uri>Elasticsearchinstance<br data-filtered="filtered">-c,--config<path>Pathtotheconfigfile<br data-filtered="filtered">-p,--port<port>Theporttobindto<br data-filtered="filtered">-q,--quietTurnsofflogging<br data-filtered="filtered">-H,--host<host>Thehosttobindto<br data-filtered="filtered">-l,--log-file<path>Thefiletologto<br data-filtered="filtered">--plugins<path>Pathtoscanforplugins<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered"></span> 将kibana放入后台运行 1 <span style= "font-family:'宋体', SimSun;" >[root@kibanakibana] #nohup./bin/kibana&<br data-filtered="filtered">[1]20765<br data-filtered="filtered">[root@kibanakibana]#nohup:ignoringinputandappendingoutputto`nohup.out'<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered"></span> 查看启动状态 1 <span style= "font-family:'宋体', SimSun;" >[root@kibanakibana] #netstat-pnutl|grep5601<br data-filtered="filtered">tcp000.0.0.0:56010.0.0.0:*LISTEN20765/./bin/../node<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered">[root@kibanakibana]#ps-ef|grepkibana<br data-filtered="filtered">root2076520572423:34pts/600:00:02./bin/../node/bin/node./bin/../src/bin/kibana.js<br data-filtered="filtered">root2078020572023:35pts/600:00:00grepkibana<br data-filtered="filtered">[root@kibanakibana]#<br data-filtered="filtered"></span> 浏览器输入http://192.168.0.44:5601/ 点击“create”创建nginx-access-log索引 切换到discover 在这里可以选择"today、Laster minutes等时间"，例如我们选择"Laster minutes"; 点击“Laster minutes” y因为我们创建的索引是在15分钟以内的，所以这里显示找不到结果，这里选择today，时间选长一点，再看 点击“today” 下面是根据需要列出以前实验的操作步骤，仅供参考 如果根据需求显示（默认是显示全部):这里用add来选择要显示的内容 搜索功能，比如搜索状态为404的 搜索状态为200的 总结ELKStack工作流程 元数据（tomcat、Apache、PHP等服务器的日志文件log）-------->logstash将原始数据写入到redis中，然后通过logstash将redis中的数据写入到elasticsearch，最后通过kibana对elasticsearch数据进行分析整理，并展示出来 补充elasticsearch配置文件注释: 1 <span style="font-family:'宋体', SimSun;">elasticsearch的config文件夹里面有两个配置文件：elasticsearch.yml和logging.yml，第一个是es的基本配置文件，第二个是日志配置文件，<br data-filtered="filtered">es也是使用log4j来记录日志的，所以logging.yml里的设置按普通log4j配置文件来设置就行了。下面主要讲解下elasticsearch.yml这个文件中可配置的东西。<br data-filtered="filtered">cluster.name:elasticsearch<br data-filtered="filtered">配置es的集群名称，默认是elasticsearch，es会自动发现在同一网段下的es，如果在同一网段下有多个集群，就可以用这个属性来区分不同的集群。<br data-filtered="filtered">node.name:"FranzKafka"<br data-filtered="filtered">节点名，默认随机指定一个name列表中名字，该列表在es的jar包中config文件夹里name.txt文件中，其中有很多作者添加的有趣名字。<br data-filtered="filtered">node.master:true<br data-filtered="filtered">指定该节点是否有资格被选举成为node，默认是true，es是默认集群中的第一台机器为master，如果这台机挂了就会重新选举master。<br data-filtered="filtered">node.data:true<br data-filtered="filtered">指定该节点是否存储索引数据，默认为true。<br data-filtered="filtered">index.number_of_shards:5<br data-filtered="filtered">设置默认索引分片个数，默认为5片。<br data-filtered="filtered">index.number_of_replicas:1<br data-filtered="filtered">设置默认索引副本个数，默认为1个副本。<br data-filtered="filtered">path.conf:/path/to/conf<br data-filtered="filtered">设置配置文件的存储路径，默认是es根目录下的config文件夹。<br data-filtered="filtered">path.data:/path/to/data<br data-filtered="filtered">设置索引数据的存储路径，默认是es根目录下的data文件夹，可以设置多个存储路径，用逗号隔开，例：<br data-filtered="filtered">path.data:/path/to/data1,/path/to/data2<br data-filtered="filtered">path.work:/path/to/work<br data-filtered="filtered">设置临时文件的存储路径，默认是es根目录下的work文件夹。<br data-filtered="filtered">path.logs:/path/to/logs<br data-filtered="filtered">设置日志文件的存储路径，默认是es根目录下的logs文件夹<br data-filtered="filtered">path.plugins:/path/to/plugins<br data-filtered="filtered">设置插件的存放路径，默认是es根目录下的plugins文件夹<br data-filtered="filtered">bootstrap.mlockall:true<br data-filtered="filtered">设置为true来锁住内存。因为当jvm开始swapping时es的效率会降低，所以要保证它不swap，可以把ES_MIN_MEM和ES_MAX_MEM两个环境变量设置成同一个值，并且保证机器有足够的内存分配给es。同时也要允许elasticsearch的进程可以锁住内存，linux下可以通过`ulimit-lunlimited`命令。<br data-filtered="filtered">network.bind_host:192.168.0.1<br data-filtered="filtered">设置绑定的ip地址，可以是ipv4或ipv6的，默认为0.0.0.0。<br data-filtered="filtered">network.publish_host:192.168.0.1<br data-filtered="filtered">设置其它节点和该节点交互的ip地址，如果不设置它会自动判断，值必须是个真实的ip地址。<br data-filtered="filtered">network.host:192.168.0.1<br data-filtered="filtered">这个参数是用来同时设置bind_host和publish_host上面两个参数。<br data-filtered="filtered">transport.tcp.port:9300<br data-filtered="filtered">设置节点间交互的tcp端口，默认是9300。<br data-filtered="filtered">transport.tcp.compress:true<br data-filtered="filtered">设置是否压缩tcp传输时的数据，默认为false，不压缩。<br data-filtered="filtered">http.port:9200<br data-filtered="filtered">设置对外服务的http端口，默认为9200。<br data-filtered="filtered">http.max_content_length:100mb<br data-filtered="filtered">设置内容的最大容量，默认100mb<br data-filtered="filtered">http.enabled:false<br data-filtered="filtered">是否使用http协议对外提供服务，默认为true，开启。<br data-filtered="filtered">gateway.type:local<br data-filtered="filtered">gateway的类型，默认为local即为本地文件系统，可以设置为本地文件系统，分布式文件系统，hadoop的HDFS，和amazon的s3服务器，其它文件系统的设置方法下次再详细说。<br data-filtered="filtered">gateway.recover_after_nodes:1<br data-filtered="filtered">设置集群中N个节点启动时进行数据恢复，默认为1。<br data-filtered="filtered">gateway.recover_after_time:5m<br data-filtered="filtered">设置初始化数据恢复进程的超时时间，默认是5分钟。<br data-filtered="filtered">gateway.expected_nodes:2<br data-filtered="filtered">设置这个集群中节点的数量，默认为2，一旦这N个节点启动，就会立即进行数据恢复。<br data-filtered="filtered">cluster.routing.allocation.node_initial_primaries_recoveries:4<br data-filtered="filtered">初始化数据恢复时，并发恢复线程的个数，默认为4。<br data-filtered="filtered">cluster.routing.allocation.node_concurrent_recoveries:2<br data-filtered="filtered">添加删除节点或负载均衡时并发恢复线程的个数，默认为4。<br data-filtered="filtered">indices.recovery.max_size_per_sec:0<br data-filtered="filtered">设置数据恢复时限制的带宽，如入100mb，默认为0，即无限制。<br data-filtered="filtered">indices.recovery.concurrent_streams:5<br data-filtered="filtered">设置这个参数来限制从其它分片恢复数据时最大同时打开并发流的个数，默认为5。<br data-filtered="filtered">discovery.zen.minimum_master_nodes:1<br data-filtered="filtered">设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1，对于大的集群来说，可以设置大一点的值（2-4）<br data-filtered="filtered">discovery.zen.ping.timeout:3s<br data-filtered="filtered">设置集群中自动发现其它节点时ping连接超时时间，默认为3秒，对于比较差的网络环境可以高点的值来防止自动发现时出错。<br data-filtered="filtered">discovery.zen.ping.multicast.enabled:false<br data-filtered="filtered">设置是否打开多播发现节点，默认是true。<br data-filtered="filtered">discovery.zen.ping.unicast.hosts:["host1","host2:port","host3[portX-portY]"]<br data-filtered="filtered">设置集群中master节点的初始列表，可以通过这些节点来自动发现新加入集群的节点。<br data-filtered="filtered">下面是一些查询时的慢日志参数设置<br data-filtered="filtered">index.search.slowlog.level:TRACE<br data-filtered="filtered">index.search.slowlog.threshold.query.warn:10s<br data-filtered="filtered">index.search.slowlog.threshold.query.info:5s<br data-filtered="filtered">index.search.slowlog.threshold.query.debug:2s<br data-filtered="filtered">index.search.slowlog.threshold.query.trace:500ms<br data-filtered="filtered">index.search.slowlog.threshold.fetch.warn:1s<br data-filtered="filtered">index.search.slowlog.threshold.fetch.info:800ms<br data-filtered="filtered">index.search.slowlog.threshold.fetch.debug:500ms<br data-filtered="filtered">index.search.slowlog.threshold.fetch.trace:200ms<br data-filtered="filtered"></span> 本文转自027ryan 51CTO博客，原文链接：http://blog.51cto.com/ucode/1775157，如需转载请自行联系原作者
