CentOS7下Elastic Stack 5.0日志分析系统搭建
一、概述 Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。 Logstash是一个开源的用于收集,分析和存储日志的工具。 Kibana也是一个开源和免费的工具，Kibana可以为Logstash和ElasticSearch提供的日志分析友好的Web界面，可以汇总、分析和搜索重要数据日志。 Beats是elasticsearch公司开源的一款采集系统监控数据的代理agent，是在被监控服务器上以客户端形式运行的数据收集器的统称，可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch，然后进行后续的数据分析活动。Beats由如下组成: 1.Packetbeat：是一个网络数据包分析器，用于监控、收集网络流量信息， Packetbeat嗅探服务器之间的流量，解析应用层协议，并关联到消息的处理， 其支持ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、 MongoDB、Memcache等协议； 2. Filebeat：用于监控、收集服务器日志文件，其已取代 logstash forwarder； 3.Metricbeat：可定期获取外部系统的监控指标信息，其可以监控、收集 Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、 Redis、System、Zookeeper等服务； 4.Winlogbeat：用于监控、收集Windows系统的日志信息； 5. Create your own Beat：自定义beat ，如果上面的指标不能满足需求，elasticsarch鼓励开发者 使用go语言，扩展实现自定义的beats，只需要按照模板，实现监控的输入，日志，输出等即可。 Beats 将搜集到的数据发送到 Logstash，经 Logstash 解析、过滤后，将其发送到 Elasticsearch 存储，并由 Kibana 呈现给用户。 Beats 作为日志搜集器没有Logstash 作为日志搜集器消耗资源，解决了 Logstash 在各服务器节点上占用系统资源高的问题。 Elastic Stack官方下载地址:https://www.elastic.co/downloads。 二、开源实时日志分析系统Elastic Stack 5.0部署: A.安装依赖包JDK ①关闭防火墙 1 2 3 4 #cat/etc/selinux/config|grep-v"#" SELINUX=disabled ##关闭selinux SELINUXTYPE=targeted #systemctlstopfirewalld##关闭防火墙 ②下载安装JDK 1 2 3 4 5 6 #wgethttp://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm #yuminstalljre##这里我只安装的是jre环境，一样可用 #java-version openjdkversion "1.8.0_111" OpenJDKRuntimeEnvironment(build1.8.0_111-b15) OpenJDK64-BitServerVM(build25.111-b15,mixedmode B.安装Elasticsearch ①下载安装elasticsearch 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch #cat>/etc/yum.repos.d/elasticsearch.repo<<EOF [elasticsearch-5.x] name=Elasticsearchrepository for 5.xpackages baseurl=https: //artifacts .elastic.co /packages/5 .x /yum gpgcheck=1 gpgkey=https: //artifacts .elastic.co /GPG-KEY-elasticsearch enabled=1 autorefresh=1 type =rpm-md EOF #yuminstallelasticsearch #systemctlstartelasticsearch #systemctlstatuselasticsearch #/usr/share/elasticsearch/bin/elasticsearch-V##查看elasticsearch版本 Version:5.0.1,Build:080bb47 /2016-11-11T22 :08:49.812Z,JVM:1.8.0_111 注:安装后出现如下问题，致使elasticsearch无法启动 1 2 3 4 5 6 7 #systemctlstatuselasticsearch OpenJDK64-BitServerVMwarning:Ifthenumberofprocessorsisexpectedtoincreasefromone, then youshouldconf...CThreads=N OpenJDK64-BitServerVMwarning:INFO:os::commit_memory(0x0000000085330000,2060255232,0)failed;error= 'Cannota...' (errno=12) #ThereisinsufficientmemoryfortheJavaRuntimeEnvironmenttocontinue. #Nativememoryallocation(mmap)failedtomap2060255232bytesforcommittingreservedmemory. #Anerrorreportfilewithmoreinformationissavedas: #/tmp/hs_err_pid11084.log 解决: 配置elasticsearch下的jvm.options： 1 2 3 4 5 #vi/etc/elasticsearch/jvm.options -Xms4g ##启用如下两项 -Xmx4g ##-Xms2g##关闭如下两项 ##-Xmx2g ②ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300。 1 #ss-tlnp|grep-E'9200|9300' ③测试服务 1 2 3 4 5 6 7 8 9 10 11 12 13 14 #curl-XGEThttp://localhost:9200 { "name" : "XVY0Ovb" , "cluster_name" : "elasticsearch" , "cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA" , "version" :{ "number" : "5.0.1" , "build_hash" : "080bb47" , "build_date" : "2016-11-11T22:08:49.812Z" , "build_snapshot" : false , "lucene_version" : "6.2.1" }, "tagline" : "YouKnow,forSearch" } 也可以使用如下命令测试 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 #curl-i-XGET'localhost:9200/' HTTP /1 .1200OK content- type :application /json ;charset=UTF-8 content-length:327 { "name" : "XVY0Ovb" , "cluster_name" : "elasticsearch" , "cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA" , "version" :{ "number" : "5.0.1" , "build_hash" : "080bb47" , "build_date" : "2016-11-11T22:08:49.812Z" , "build_snapshot" : false , "lucene_version" : "6.2.1" }, "tagline" : "YouKnow,forSearch" } C.安装Logstash 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch #cat>/etc/yum.repos.d/logstash.repo<<EOF [logstash-5.x] name=Elasticrepository for 5.xpackages baseurl=https: //artifacts .elastic.co /packages/5 .x /yum gpgcheck=1 gpgkey=https: //artifacts .elastic.co /GPG-KEY-elasticsearch enabled=1 autorefresh=1 type =rpm-md EOF #yuminstalllogstash #systemctlstartlogstash #systemctlstatuslogstash #/usr/share/logstash/bin/logstash-V##查看logstash版本 logstash5.0.1 D.安装Kibana ①安装Kibana 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch #cat>/etc/yum.repos.d/kibana.repo<<EOF [kibana-5.x] name=Kibanarepository for 5.xpackages baseurl=https: //artifacts .elastic.co /packages/5 .x /yum gpgcheck=1 gpgkey=https: //artifacts .elastic.co /GPG-KEY-elasticsearch enabled=1 autorefresh=1 type =rpm-md EOF #yuminstallkibana #systemctlstartkibana #systemctlstatuskibana #/usr/share/kibana/bin/kibana-V##查看kibana版本 5.0.1 ②只需更改如下配置 1 2 #cat/etc/kibana/kibana.yml|grep-v"#" server.host: "192.168.147.128" ③检测服务 1 #ss-tlnp|grep5601##Kibana默认进程名：node，端口5601 浏览器输入http://localhost:5601 E、Beats安装部署 a.安装部署Filebeat ①安装Filebeat 1 2 3 4 5 6 #curl-L-Ohttps://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.1-x86_64.rpm #rpm-ivhfilebeat-5.0.1-x86_64.rpm #systemctlstartfilebeat #systemctlstatusfilebeat #filebeat.sh-version##查看版本 filebeatversion5.0.1(amd64),libbeat5.0.1 ②配置Filebeat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #cd/etc/filebeat #cpfilebeat.ymlfilebeat.yml.bak #vi/etc/filebeat/filebeat.yml##配置filebeat #=============Filebeatprospectors=============== filebeat.prospectors: -input_type:log paths: - /var/log/ *.log #====================Outputs===================== #-------------Elasticsearchoutput--------------- output.elasticsearch: #Arrayofhoststoconnectto. hosts:[ "localhost:9200" ] #----------------Logstashoutput----------------- output.logstash: #TheLogstashhosts hosts:[ "localhost:5043" ] ##只需配置该处，其他默认即可 #filebeat.sh-configtest-e##验证配置文件 ③配置Logstash 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 #cat/etc/logstash/conf.d/logstash.conf input{ beats{ port=> "5043" } } filter{ grok{ match=>{ "message" => "%{COMBINEDAPACHELOG}" } } geoip{ source => "clientip" } } output{ elasticsearch{ hosts=>[ "localhost:9200" ] } } #systemctlrestartlogstash##重启logstash #/usr/share/logstash/bin/logstash-f/etc/logstash/conf.d/logstash.conf--config.test_and_exit##验证配置文件 SendingLogstashlogsto /var/log/logstash which isnowconfiguredvialog4j2.properties ConfigurationOK #ss-tlnp|grep-E'5043|9600' ④配置kibana 浏览器输入http://localhost:5601，配置filebeat的索引（只需输入filebeat-*即可）。 在第一个框里输入filebeat-*后稍等片刻，kibana会自动识别，OK后下面的按钮会由灰色变为可操控的按钮"Create",如上图所示。点击该按钮后，最后就会呈现如下图所示: 我们再回过头新建logstash的索引，浏览器输入http://localhost:5601，点击左边栏的”Management” ===> 然后点击“index Patterns” ===> ===> 然后点击“Add New” ===> ===> 点击“Crete”按钮创建logstash索引，创建完成后即会展现如下图所示: b.安装部署Packetbeat ①安装配置Packetbeat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 #yuminstalllibpcap #curl-L-Ohttps://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.0.1-x86_64.rpm #rpm-ivhpacketbeat-5.0.1-x86_64.rpm #cat/etc/packetbeat/packetbeat.yml ====================Networkdevice=================== packetbeat.interfaces.device:any ##捕获所有消息发送或接收的网络接口 ========================Flows======================== packetbeat.flows: timeout:30s period:10s ==================Transactionprotocols============== ##如下是packetbeat默认支持的主要协议及端口 packetbeat.protocols.icmp: enabled: true packetbeat.protocols.amqp: ports:[5672] packetbeat.protocols.cassandra: ports:[9042] packetbeat.protocols.dns: ports:[53] include_authorities: true include_additionals: true packetbeat.protocols.http: ports:[80,8080,8000,5000,8002] packetbeat.protocols.memcache: ports:[11211] packetbeat.protocols.mysql: ports:[3306] packetbeat.protocols.pgsql: ports:[5432] packetbeat.protocols.redis: ports:[6379] packetbeat.protocols.thrift: ports:[9090] packetbeat.protocols.mongodb: ports:[27017] packetbeat.protocols.nfs: ports:[2049] =========================General========================= =========================Outputs========================= -------------------Elasticsearchoutput------------------ output.elasticsearch: hosts:[ "localhost:9200" ] ---------------------Logstashoutput--------------------- output.logstash: hosts:[ "localhost:5043" ] ##只需配置该处，其他默认即可 =============================Logging===================== ②验证配置并启动packetbeat 1 2 3 4 5 6 7 #packetbeat.sh-version##查看packetbeat版本 packetbeatversion5.0.1(amd64),libbeat5.0.1 #packetbeat.sh-configtest-e##测试配置文件 ...... ConfigOK #systemctlstartpacketbeat #systemctlstatuspacketbeat ③配置Kibana(新建packetbeat索引) 在http://localhost:5601下新建索引页面输入“packetbeat-*”，之后kibana会自动更新，在“Time-field name”下面的三个选项中选择“@timestamp”，最后点击“Create”创建即可。 创建完成后，kibana显示如下: c.安装部署Metricbeat ①安装配置metricbeat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 #curl-L-Ohttps://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.1-x86_64.rpm #rpm-ivhmetricbeat-5.0.1-x86_64.rpm #cat/etc/metricbeat/metricbeat.yml =================Modulesconfiguration================= metricbeat.modules: ----------------------SystemModule--------------------- -module:system metricsets: -cpu -load -filesystem -fsstat -memory -network -process enabled: true period:10s processes:[ '.*' ] =========================General======================= =========================Outputs======================= -------------------Elasticsearchoutput---------------- output.elasticsearch: hosts:[ "localhost:9200" ] ---------------------Logstashoutput------------------- output.logstash: hosts:[ "localhost:5043" ] ##只需配置该项，其他默认即可 =======================Logging========================= ②验证配置并启动metricbeat 1 2 3 4 5 #metricbeat.sh-version##查看版本信息 metricbeatversion5.0.1(amd64),libbeat5.0.1 #metricbeat.sh-configtest-e##验证配置文件 #systemctlstartmetricbeat #systemctlstatusmetricbeat ③配置kibana(新建metricbeat索引) 在http://localhost:5601下新建索引页面输入“metricbeat-*”，之后kibana会自动更新，在“Time-field name”下面的选项中选择“@timestamp”，最后点击“Create”创建即可。 最后呈现如下图所示: 注1: 关于ELK Stack的一些查询语句: ①查询filebeat 1 #curl-XGET'http://localhost:9200/filebeat-*/_search?pretty' ②查询packetbeat 1 #curl-XGET'http://localhost:9200/packetbeat-*/_search?pretty' ③查询metricbeat 1 #curl-XGET'http://localhost:9200/metricbeat-*/_search?pretty' ④查询集群健康度 1 #curl'localhost:9200/_cat/health?v' ⑤查看节点列表 1 2 3 #curl'localhost:9200/_cat/nodes?v' ipheap.percent ram .percentcpuload_1mload_5mload_15mnode.rolemastername 127.0.0.1379330.050.070.41mdi*XVY0Ovb ⑥列出所有索引 1 2 3 4 5 6 7 8 9 10 11 12 #curl'localhost:9200/_cat/indices?v' healthstatusindexuuidprirepdocs.countdocs.deletedstore.sizepri.store.size yellow open filebeat-2016.11.28Mn4MzxdTRaCj9iseutcmqA512012kb12kb yellow open filebeat-2016.11.29iMrr710mT42mApxdV62k-A51159065.9kb65.9kb yellow open packetbeat-2016.11.29wkTcIwD6RgiiCFwlWBIILA51565201.6mb1.6mb yellow open customerNvxXLgHoREefJLRhot13Ug5100800b800b yellow open packetbeat-2016.11.28Beoe07S7QB-dntNV4nxJNQ5124460676.4kb676.4kb yellow open test M7WbkYq2QNmeJ9NOyMfMZA5100800b800b yellow open logstash-2016.11.28pcb_84ChSBe9A7VRd-SQNw511610123.2kb123.2kb yellow open metricbeat-2016.11.29AmVeT1xCQGCnxlAFXUxhYw5194459037.6mb37.6mb yellow open logstash-2016.11.296PCKMYKCSVmPfdg-Sx2ARA5185772020.3mb20.3mb yellow open .kibanaQYTg0I5KS-yc3d7GSey3Zw1150102kb102kb 注2: 如果搭建期间有什么不清楚或不了解的，建议去看官方文档，文档地址如下: https://www.elastic.co/guide/index.html 注3:如需创建自己的beat，该处有教程可供学习: http://www.itnose.net/detail/6675459.html 本文转自 结束的伤感 51CTO博客，原文链接:http://blog.51cto.com/wangzhijian/1878636
