go-stash 是一个高效的从 Kafka 获取，根据配置的规则进行处理，然后发送到 ElasticSearch 集群的工具。
go-stash 有大概 logstash 5 倍的吞吐性能，并且部署简单，一个可执行文件即可。
![]()
安装
cd stash && go build stash.go
Quick Start
./stash -f etc/config.yaml
docker run -d -v `pwd`/etc:/app/etc kevinwan/go-stash
config.yaml示例如下:
Clusters:
- Input:
Kafka:
Name: go-stash
Log:
Mode: file
Brokers:
- "172.16.48.41:9092"
- "172.16.48.42:9092"
- "172.16.48.43:9092"
Topic: ngapplog
Group: stash
Conns: 3
Consumers: 10
Processors: 60
MinBytes: 1048576
MaxBytes: 10485760
Offset: first
Filters:
- Action: drop
Conditions:
- Key: status
Value: 503
Type: contains
- Key: type
Value: "app"
Type: match
Op: and
- Action: remove_field
Fields:
- message
- source
- beat
- fields
- input_type
- offset
- "@version"
- _score
- _type
- clientip
- http_host
- request_time
Output:
ElasticSearch:
Hosts:
- "http://172.16.188.73:9200"
- "http://172.16.188.74:9200"
- "http://172.16.188.75:9200"
Index: "go-stash-{{yyyy.MM.dd}}"
MaxChunkBytes: 5242880
GracePeriod: 10s
Compress: false
TimeZone: UTC
ES性能写入测试
测试环境
- stash服务器：3台 4核 8G
- es服务器： 15台 16核 64G
关键配置
- Input:
Conns: 3
Consumers: 10
Processors: 60
MinBytes: 1048576
MaxBytes: 10485760
Filters:
- Action: remove_field
Fields:
- message
- source
- beat
- fields
- input_type
- offset
- request_time
Output:
Index: "nginx_pro-{{yyyy.MM.d}}"
Compress: false
MaxChunkBytes: 5242880
TimeZone: UTC
写入速度平均在15W/S以上
![]()
