Logstash5.3借助临时字段修改@timestamp为北京时间，方便按天生成output文件

小程序：扫一扫查出行
【扫一扫了解最新限行尾号】
复制小程序

$more config/first-pipeline.conf
input {
beats {
port => ""
}
}
filter {
if [type] == "speech" {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))"
}
} if [type] == "speech-en" {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))"
} } if [type] == "client-agent" {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))"
} } if [type] == "client-agent-en" {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))"
} } if [type] == "session-manager" {
ruby {
code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('timestamp'))"
}
}
}
output {
stdout {
codec => rubydebug
}
file{
codec => line {format => "%{message}"}
path => "/home/baoshan/elk/data/logstash/%{type}.%{+YYYYMMdd}"
}
# elasticsearch {
# hosts => ["test41:9200","test42:9200","test43:9200"]
# index => "%{hostabc}"
# document_type => "%{hostabc}"
# #protocol: "http"
# flush_size =>
# idle_flush_time =>
# user => "elastic"
# password => "baoshan"
# }
}

核心代码为ruby中的code语句。（感觉这是最笨的方法，继续探求精简的办法。。。。）

特么的找了一天了，现在大部分都还不是logstash5.x的

下面这种方法试烂了都不管用，不知哪里不对，还请高手指教：

grok {
match => {
"message" => "time%{NUMBER:timestamp}id%{UUID:sn}asr%{NOTSPACE:asr}nlp%{NOTSPACE:nlp}domain%{NOTSPACE:domain}intent%{NOTSPACE:intent}"
}
}
date {
match => ["timestamp", "UNIX_MS"] #因为我的日志时间戳为UNIX时间戳，毫秒级，后来发现这个时间戳硬生生被ELK改成了UTC时间
target => "@timestamp"
locale => "en"
timezone => "+00:00"
}

所以有了下面的配置

改配置文件包括两个知识点

1. 不可见字符（ctrl+A，ctrl+B）grok的方法

2. logstash时间戳@timestamp修改为日志中时间字段的方法

input {
beats {
port => ""
}
}
filter {
grok {
match => [ # 此处的^A为vim下的CTRL+A
"message", "time\^B%{INT:timestamp}\^Aid\^B%{NOTSPACE:sn}\^Aasr\^B%{NOTSPACE:asr}\^Anlp\^B%{DATA:nlp}\^Adomain\^B%{JAVACLASS:domain}\^Aintent\^B%{NOTSPACE:intent}"
]
}
date {
match => ["timestamp", "UNIX_MS"]
target => "@timestamp"
}
ruby {
code => "event.set('temp', event.get('@timestamp').time.localtime + 8*60*60); event.set('@timestamp', event.get('temp'))"
}
}
output {
# stdout { codec => rubydebug }
file {
codec => line {format => "%{message}"}
path => "/home/admin/data/speech/speech.log.%{+YYYYMMdd}"
}
file {
codec => line {format => "%{+YYYY-MM-dd HH:mm:ss}^A%{sn}^A%{asr}^A%{nlp}^A%{domain}^A%{intent}"}
path => "/home/admin/data/speech/speech%{+YYYY-MM-dd}"
}
}