logstash简介及基本操作
logstash常用于日志系统中做日志采集设备，最常用于ELK中作为日志收集器使用
logstash的基本流程架构：input | filter | output 如需对数据进行额外处理，filter可省略。
用于在将event通过output发出之前对其实现某些处理功能。grok。
grok：用于分析结构化文本数据。目前 是logstash中将非结构化数据日志数据转化为结构化的可查询数据的不二之选
3.3 Output(输出）：将我们过滤出的数据保存到那些数据库和相关存储中， 。
inpust：必须，负责产生事件（Inputs generate events），常用：File、syslog、redis、beats（如：Filebeats）
filters：可选，负责数据处理与转换（filters modify them），常用：grok、mutate、drop、clone、geoip
outpus：必须，负责数据输出（outputs ship them elsewhere），常用：elasticsearch、file、graphite、statsd
input { 从哪个地方读取，输入数据。
}
filter { 依据grok模式对数据进行分析结构化
}
output { 将分析好的数据输出存储到哪些地方
}
2.1我们自定义gork模式对日志进行过滤。
语法格式：
%{SYNTAX:SEMANTIC}
SYNTAX：预定义模式名称；
SEMANTIC：匹配到的文本的自定义标识符；
[root@node1 conf.d]# vim groksimple.conf
input {
stdin {}
}
filter {
grok {
match => { "message" => "%{IP:clientip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}
output {
stdout {
codec => rubydebug
}
}
[root@node1 conf.d]# logstash -f /etc/logstash/conf.d/groksimple.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-10-13 14:29:41.936 [main] runner - Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.262-b10 on 1.8.0_262-b10 +indy +jit [linux-x86_64]"}
[WARN ] 2020-10-13 14:29:42.412 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-10-13 14:29:44.025 [Converge PipelineAction::Create<main>] Reflections - Reflections took 42 ms to scan 1 urls, producing 22 keys and 45 values
[INFO ] 2020-10-13 14:29:44.995 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/groksimple.conf"], :thread=>"#<Thread:0x4ca2f74b run>"}
[INFO ] 2020-10-13 14:29:45.749 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.74}
[INFO ] 2020-10-13 14:29:45.820 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2020-10-13 14:29:45.902 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-10-13 14:29:46.098 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
1.1.1.1 get /index.html 30 0.23 我们标准输入一些日志信息。
{
"@timestamp" => 2020-10-13T06:30:11.973Z,
"host" => "node1",
"@version" => "1",
"request" => "/index.html",
"message" => "1.1.1.1 get /index.html 30 0.23",
"duration" => "0.23",
"clientip" => "1.1.1.1",
"method" => "get",
"bytes" => "30"
}
推荐阅读：
https://blog.csdn.net/yurun_house/article/details/109025588
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/69949806/viewspace-2903536/ ，如需转载，请注明出处，否则将追究法律责任。
