我所期望达到的效果，就是每个日期一条，在kibana界面也是一条。多行异常的也是一条。
其实，很简单，就是加个反向判断。
logstash原理
一个客户端，一个服务器，就是这样的模式
没什么神奇的，最麻烦的正则匹配，不好弄。听说storm更好用。
1.client配置
cat /etc/logstash/conf.d/shipper.conf
input {
file {
path => ["/opt/src/logs/*/*/*/*"]
type => "service"
start_position => "beginning"
}
}
filter {
if [type] == "service" {
multiline {
patterns_dir => "/etc/logstash/conf.d"
pattern => "(^%{MYTIMESTAMP})"
negate => true
what => "previous"
}
}
grok {
patterns_dir => "/etc/logstash/conf.d"
match => [ "message", "%{MYLOG}" ]
add_field => [ "log_ip", "随便写" ]
}
}
output {
stdout {}
redis {
host => "你的服务端ip"
port => 6379
password => "xx"
data_type => "list"
key => "key_count"
}
}
patterns_dir => "/etc/logstash/conf.d" 这个东西就是正则表达式
新建一个文件 cat /etc/logstash/conf.d/j2ee
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
HTTPPORT ([a-zA-Z$_0-9]*\-)*([0-9])
JAVALOGMESSAGE (.*)
MYTIMESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{HOUR}:%{MINUTE}:%{SECOND}
MYLOG %{MYTIMESTAMP:mytimestamp}\s\[%{HTTPPORT:port}\]\s%{LOGLEVEL:level}\s%{JAVACLASS:class}\s-%{JAVALOGMESSAGE:logmessage}
我的日志格式是这样的
2016-10-2015:52:01.174 [http-apr-8282-exec-4] INFO c.e.w.c.... -resp:
如果你的日志不一样，那么j2ee文件要做相应的修改
http://grokdebug.herokuapp.com/ 这个网站可以测试
比如
![wKiom1gIjtTw1-TzAABvblyuSNE276.png-wh_50]( "搜狗截图20161020172837.png")
启动脚本logstash.sh
#!/bin/bash
. /etc/init.d/functions
function start(){
cd /home/
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/shipper.conf & >>/dev/null 2>&1
}
function stop(){
sudo kill -9 `ps -ef|grep logstash|grep -v grep|awk '{print $2}'`
#for i in `ps -ef|grep logstash|grep -v grep|awk '{print $2}'`
#do;sudo kill -9 $i;done
}
case $1 in
start)
start
;;
stop)
stop
;;
*)
printf "Usage sh *.sh start|stop"
;;
esac
sh logstash.sh stop/start
2.server配置
cat /etc/logstash/conf.d/indexer.conf
input {
redis {
host => "你的服务端ip"
port => 6379
password => "xx"
type => "redis-input"
data_type => "list"
key => "key_count"
}
}
output {
stdout {}
elasticsearch {
cluster => "elasticsearch"
codec => "json"
protocol => "http"
}
}
3.kibana效果
![wKioL1gOuiaRdfbuAADZlql089g587.png-wh_50]( "搜狗截图20161025094901.png")
多行为一行就不截图了
注意有事启动了logstash，但是没有日志传送过来
-
input file 匹配类型，正则有问题
-
filter正则有问题
总而言之，就是配置有问题。
demo
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
|
client
cat
demo.conf
input {
file
{
path => [
"/home/python/demo/*/*/*/*"
]
type
=>
"demo"
start_position =>
"beginning"
}
}
filter {
if
[
type
] ==
"demo"
{
multiline {
patterns_dir =>
"/etc/logstash/conf.d/patterns"
pattern =>
"(^%{MYTIMESTAMP})"
negate =>
true
what =>
"previous"
}
}
grok {
patterns_dir =>
"/etc/logstash/conf.d/patterns"
match => [
"message"
,
"%{MYLOG}"
]
add_field => [
"log_ip"
,
"172.29.xx.xx"
]
}
}
output {
stdout {}
redis {
host =>
"172.29.xx.xx"
port => 6379
password =>
"xxxxxx"
data_type =>
"list"
key =>
"key_count"
}
}
cd
/etc/logstash/conf
.d
/patterns
cat
j2ee
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
HTTPPORT ([a-zA-Z$_0-9]*\-)*([0-9])
JAVALOGMESSAGE (.*)
MYTIMESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}
MYLOG %{MYTIMESTAMP:mytimestamp}\s\[%{HTTPPORT:port}\]\s%{LOGLEVEL:level}\s%{JAVACLASS:class}\s-%{JAVALOGMESSAGE:logmessage}
logstash -f
/etc/logstash/conf
.d
/logstash-indexer
.conf or service logstash start
server
nohup
/usr/share/elasticsearch/bin/elasticsearch
&
nohup
/usr/bin/kibana
&
logstash -f
/etc/logstash/conf
.d
/logstash-indexer
.conf or service logstash start
|
还有es+gra+logstash
http://play.grafana.org/dashboard/db/elasticsearch-metrics doc
本文转自 liqius 51CTO博客，原文链接：http://blog.51cto.com/szgb17/1863975，如需转载请自行联系原作者
