安装过程不表.

年初接手了搜索项目.
如何使用ELK统计用户正在搜索频率最高的歌曲?
如何使用ELK统计每分钟的访问量?

resin accesslog 如下:
192.168.216.25 - "null-user" [17/May/2016:21:13:49 +0800] "POST /search/dispatch.htm HTTP/1.1" 200 6302 "-" "Apache-HttpClient/4.2.2 (java 1.5)" "192.168.227.61:8101" - 99000
192.168.216.29 - "null-user" [17/May/2016:21:13:49 +0800] "POST /search/dispatch.htm HTTP/1.1" 200 5825 "-" "Apache-HttpClient/4.2.2 (java 1.5)" "192.168.227.61:8101" - 48000
192.168.216.29 - "null-user" [17/May/2016:21:13:49 +0800] "POST /search/dispatch.htm HTTP/1.1" 200 6091 "-" "Apache-HttpClient/4.2.2 (java 1.5)" "192.168.227.61:8101" - 57000
192.168.216.25 - "null-user" [17/May/2016:21:13:50 +0800] "POST /search/dispatch.htm HTTP/1.1" 200 6303 "-" "Apache-HttpClient/4.2.2 (java 1.5)" "192.168.227.61:8101" - 73000
192.168.216.29 - "null-user" [17/May/2016:21:13:50 +0800] "POST /search/dispatch.htm HTTP/1.1" 200 6449 "-" "Apache-HttpClient/4.2.2 (java 1.5)" "192.168.227.61:8101" - 59000
192.168.216.25 - "null-user" [17/May/2016:21:13:50 +0800] "POST /search/sag.htm HTTP/1.1" 200 777 "-" "Apache-HttpClient/4.2.2 (java 1.5)" "192.168.227.61:8101" - 2000

resin 日志如下:
2016-05-17 21:16:51,240 - INFO -4551312890 - [resin-port-8101-37806]- com.search.controller.SearchCtrl - - Time_KSC 祝你生曰快乐 result size: 39447 cost time: 63
2016-05-17 21:16:51,267 - INFO -4551312917 - [resin-port-8101-3009]- com.kit.service.impl.SagArtistServiceImpl - - SagArtistServiceImplLog word : 舞 resultsNum: 116
2016-05-17 21:16:51,267 - INFO -4551312917 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog search Song Trie cost time : 0ms
2016-05-17 21:16:51,270 - INFO -4551312920 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog word : 舞 resultsNum: 2737
2016-05-17 21:16:51,276 - INFO -4551312926 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog insert into treemap cost time : 6ms
2016-05-17 21:16:51,276 - INFO -4551312926 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog highlight cost time : 0ms
2016-05-17 21:16:51,276 - INFO -4551312926 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog search Song Trie cost time : 0ms
2016-05-17 21:16:51,279 - INFO -4551312929 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog word : 舞 resultsNum: 2737
2016-05-17 21:16:51,282 - INFO -4551312932 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog insert into treemap cost time : 3ms
2016-05-17 21:16:51,282 - INFO -4551312932 - [resin-port-8101-3009]- com.kit.service.impl.SagSongServiceImpl - - SagSongServiceImplLog highlight cost time : 0ms
2016-05-17 21:16:51,282 - INFO -4551312932 - [resin-port-8101-3009]- com.kit.controller.SagCtrl - - SagCtrl-sag.htm cost time : 16ms
2016-05-17 21:16:51,507 - INFO -4551313157 - [resin-port-8101-37605]- com.caucho.env.thread2.ResinThread2 - - SearchServiceImpl-searchByPage search initial word 梦中的妈妈
2016-05-17 21:16:51,507 - INFO -4551313157 - [resin-port-8101-37605]- com.caucho.env.thread2.ResinThread2 - - SearchServiceImpl-searchByPage search word after process梦中的妈妈
2016-05-17 21:16:51,507 - INFO -4551313157 - [resin-port-8101-37605]- com.caucho.env.thread2.ResinThread2 - - SearchServiceImpl-searchByPage word analyzer results：梦中的妈妈
2016-05-17 21:16:51,507 - INFO -4551313157 - [resin-port-8101-37605]- com.search.util.BoostUtil - - BoostUtil-getBoost 梦中的妈妈 miss

其中SearchServiceImpl-searchByPage search initial word 表示用户输入的原始搜索关键字

logstash配置如下:

input{
file{
path=>["/home/www_op/resin/logs/lucene.access.log"]
type=>"lucene_accesslog"
}
file{
path=>["/home/www_op/resin/logs/lucene_service.log"]
type=>"lucene_service"
}
}
filter{
if [type] == "lucene_accesslog" {
grok{
match=>{
"message"=>"%{IPORHOST:clientip} - \"%{DATA:remoteUser}\" \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\" %{NUMBER:httpCode:int} (?:-|%{NUMBER:resultContentLength:int}) \"-\" \"%{DATA:UserAgent}\" \"%{DATA:host}\" - %{NUMBER:requestTime:int}"
}
add_tag=>["lucene_accesslog"]
remove_field=>["message"]
}
} else if [type] == "lucene_service" {
grok{
match=>{"message"=>"ERROR|WARNING|INFO"}
add_tag=>["ERROR"]
}
grok{
match=>{"message"=>"%{TIMESTAMP_ISO8601:timestamp} - %{WORD:loglevel}.*SearchServiceImpl-searchByPage search initial word %{GREEDYDATA:wd}"}
remove_field=>["message"]
remove_tag=>["_grokparsefailure","ERROR"]
add_tag=>["userSearchKeyWord"]
tag_on_failure => []
}
}
}
output{
if "_grokparsefailure" not in [tags]{
elasticsearch{
hosts=>["192.168.16.105:9200","192.168.16.106:9200","192.168.16.108:9200","192.168.16.109:9200","192.168.16.110:9200"]
index=>"logstash-%{type}-%{+YYYY.MM.dd}"
document_type=>"%{[tags]}"
workers=>5
template_overwrite=>true
}
}
}

关于resin日志,ERROR WARNING和INFO级别的日志均要收集.
其中涉及用户输入的原始关键词,则修改tag为userSearchKeyWord ,并且设置 tag_on_failure为null。否则前面匹配的行,由于在这里不匹配,最后会被丢弃.

启用logstash,然后在Kibana中配置一下

ok 我们的用户现在实时都在搜索什么歌曲,搜索服务的访问次数,一目了然.

对于我来说,难度最大的就是 grok 正则匹配.

有几个网站,非常有用

http://grokconstructor.appspot.com/do/match#result
https://grokdebug.herokuapp.com/
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

ITPub博客

ELK尝试

最新文章