用Kibana+Logstash+Elasticsearch快速搭建实时日志查询、收集与分析系统
安装环境 先看看都需要安装什么软件包 ruby 运行Kibana 必须， rubygems 安装ruby扩展必须 bundler 功能类似于yum JDK 运行java程序必须 redis 用来处理日志队列 logstash 收集、过滤日志 ElasticSearch 全文搜索服务(logstash集成了一个) kibana 页面展示 192.168.18.240 logstash index，kibana，JDK 192.168.18.241 logstash agent，JDK 192.168.18.242 redis 192.168.18.243 ElasticSearch，JDK 先安装redis (192.168.18.242) # wget http://redis.googlecode.com/files/redis-2.6.12.tar.gz # tar zxvf redis-2.6.12.tar.gz # mv redis-2.6.12 redis # cd redis # make -j24 # make install # vi /root/soft/redis/redis.conf 修改level为 loglevel verbose # redis-server /root/soft/redis/redis.conf & 看看 redis服务的状态 # lsof-i:6379 安装 elasticsearch (192.168.18.243) elasticsearch会依赖于java # vi /etc/apt/sources.list deb http: //ftp.debian.org/debian/ squeeze main non-free deb-src http: //ftp.debian.org/debian/ squeeze main non-free # apt-get update # apt-cache search sun-java # apt-get install sun-java6-jdk sun-java6-jre # java -version # wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.6.zip # unzip elasticsearch-0.20.6.zip # mv elasticsearch-0.20.6 /usr/local/share/elasticsearch # cd /usr/local/share/elasticsearch/bin/ # ./elasticsearch -f 在 logstash index上安装基础的软件环境： (192.168.18.240) # vi /etc/apt/sources.list deb http: //ftp.debian.org/debian/ squeeze main non-free deb-src http: //ftp.debian.org/debian/ squeeze main non-free # apt-get update # apt-cache search sun-java # apt-get install sun-java6-jdk sun-java6-jre # java -version 开始安装logstash ( 其实logstash就是一个java脚本，不需要安装...下载即用 ) # wget https://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar # vi /root/soft/redis.conf input { redis { host => '192.168.18.242' data_type => 'list' port => "6379" key => 'logstash:redis' type => 'redis-input' } } output { elasticsearch { host => '192.168.18.243' port => "9300" } } # java -jar /root/soft/logstash-1.1.9-monolithic.jar agent -f /root/soft/redis.conf -- web --backend elasticsearch:///?local 现在可以通过浏览器访问一下 http://192.168.18.240:9292 看看logstash是的页面是个什么样子 配置logstash的agent (192.168.18.241) 安装sun-java6-jre sun-java6-jdk # wget https://logstash.objects.dreamhost.com/release/logstash-1.1.9-monolithic.jar # vi /root/soft/redis.conf input { stdin { type => "stdin-type" } file { type => "linux-syslog" # Wildcards work, here :) path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ] } } output { redis { host => '192.168.18.242' data_type => 'list' key => 'logstash:redis' } } # java -jar /root/soft/logstash-1.1.9-monolithic.jar agent -f /root/soft/redis.conf & OK，最后就是 Kibana了 ，我把Kibana装在了 logstash index上面 下载地址为http://kibana.org/intro.html # apt-get install ruby rubygems # gem install bundler #bundleinstall ( /var/lib/gems/1.8/bin/bundle install ) 以上为ruby运行环境 wget https://github.com/rashidkpc/Kibana/archive/v0.2.0.tar.gz #tar zxvfKibana-0.2.0.tar.gz #cdKibana-0.2.0 直接安装就好了，非常简单，因为之前咱们已经安装好了bundle 编辑配置文件，指定elasticsearch的位置 [192.168.18.240 root@nodec:/soft/Kibana-0.2.0] #vimKibanaConfig.rb ..... Elasticsearch="192.168.18.243:9200" KibanaPort=5601 KibanaHost='0.0.0.0' ..... 主要是这几个参数 启动的话需要ruby [192.168.18.240 root@nodec:/soft/Kibana-0.2.0] #/usr/bin/rubykibana.rb& [192.168.18.240 root@nodec:/soft/Kibana-0.2.0] #==Sinatra/1.3.5hastakenthestageon5601fordevelopmentwithbackupfromThin >>Thinwebserver(v1.5.0codenameKnife) >>Maximumconnectionssetto1024 >>Listeningon0.0.0.0:5601,CTRL+Ctostop 如果ruby的东西都不缺的话，启动会很顺利，ok现在看看5601端口的状态 [192.168.233.128root@nodec:/soft/Kibana-0.2.0] #lsof-i:5601 COMMANDPIDUSERFDTYPEDEVICESIZENODENAME ruby3116root5uIPv428947TCP*:esmagent(LISTEN) 访问一下 试试看http://192.168.18.240:5601尝试搜索一下php的错误日志，比如mysql 呵呵，要的就是这个效果，日志会实时的汇总到 logstash index 上供我们查询，当然这只是开始使用logstash的第一步而已，更多的高级功能可以看看官方文档http://logstash.net/docs/1.1.9/
