ITPub博客

首页 > IT基础架构 > > ELK-基础系列(五)-Logstash 正则实践

ELK-基础系列(五)-Logstash 正则实践

作者:格伯纳 时间:2020-05-29 11:50:35 0 删除 编辑

Logstash 正则参考地址

参考地址:

应用日志文件内容

Sep  2 16:00:01 cc-prd-3-tk-13-pcs1 CRON[10590]: (root) CMD (   /usr/sbin/ntpdate -u ntp1.aliyun.com ntp2.aliyun.com >/dev/null 2>&1)
Sep  2 16:00:18 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:18 2019: PCS: recv request 339773 from ip: 27.102.113.76:55534, public_ip: 223.104.145.136
Sep  2 16:00:18 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:18 2019: PCS: 223.104.145.136:24533 223.104.145.136 00003b9f21835d46c2d36def87b223502dddf60ecb182032d20f5d3bc32124c2fac8859c873db6 e587ec4269d2ff63a25b0dc3716e63f9 5c611e85bfd0d6c50c906dfded565a0c 1.0.6.020 0 1567433717 1567440017 1 1 26 1058673192
Sep  2 16:00:18 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:18 2019: PCS: recv request, app_id: app_1w5B6O4R2o1k881k12177, ip: 223.104.145.136, os: 0, device_id: 02c3864050502d43dc514905133bcc9c
Sep  2 16:00:18 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:18 2019: PCS: allocate AN, app_id: app_1w5B6O4R2o1k881k12177, ip: 223.104.145.136, AN: an_1z566D44161870620, region: cn-east
Sep  2 16:00:18 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:18 2019: PCS: end client request 339773
Sep  2 16:00:18 cc-prd-3-tk-13-pcs1 systemd-journald: Forwarding to syslog missed 12 messages.
Sep  2 16:00:27 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:27 2019: PCS: recv request 339781 from ip: 34.92.148.220:33430, public_ip: 119.251.248.84
Sep  2 16:00:27 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:27 2019: PCS: 119.251.248.84:1420 119.251.248.84 00005435801f5d3788f16b0f2dc1f8dd794d5e06410ba8a5e7177559ca45d933dcefedd0ff2f4e e587ec4269d2ff63a25b0dc3716e63f9 5c611e85bfd0d6c50c906dfded565a0c 1.0.6.020 0 1567440027 1567440026 0 1 0 1345592700
Sep  2 16:00:27 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:27 2019: PCS: recv request, app_id: app_1w5B6O4R2o1k881k12177, ip: 119.251.248.84, os: 0, device_id: a658a1ea82ca183d10856399bba7d3ca
Sep  2 16:00:27 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:27 2019: HBase got exception THRIFT_EAGAIN (timed out)
Sep  2 16:00:27 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:27 2019: PCS: allocate AN, app_id: app_1w5B6O4R2o1k881k12177, ip: 119.251.248.84, AN: an_1b5m6i42161870620, region: cn-north
Sep  2 16:00:27 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:27 2019: PCS: end client request 339781
Sep  2 16:00:44 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:44 2019: mq(PC-cc-prd-3-tk-13-pcs1) buffer delivered: CMD_PUSH_CONN_STATUS#015#012{#012   "access-node" : [#012      {#012         "node-name" : "an_1P5n6R6k973774022",#012         "nr_cur_clients" : 0,#012         "nr_cur_conns" : 0#012      },#012      {#012         "node-name" : "an_1H5X6e6F726082009",#012         "nr_cur_clients" : 6,#012         "nr_cur_conns" : 7#012      },#012      {#012         "node-name" : "an_1z566D44161870620",#012         "nr_cur_clients" : 4,#012         "nr_cur_conns" : 6#012      },#012      {#012         "node-name" : "an_1b5m6i42161870620",#012         "nr_cur_clients" : 5,#012         "nr_cur_conns" : 6#012      }#012   ],#012   "app-list" : [#012      {#012         "app_id" : "app_1w5B6O4R2o1k881k12177",#012         "reject_new_conn" : 0#012      }#012   ]#012}
Sep  2 16:01:30 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:01:30 2019: PCS: recv request 339841 from ip: 35.194.199.109:48494, public_ip: 27.18.224.156
Sep  2 16:01:30 cc-prd-3-tk-13-pcs1 systemd-journald: Forwarding to syslog missed 8 messages.
Sep  2 16:01:30 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:01:30 2019: PCS: 27.18.224.156:26775 27.18.224.156 00004713c0715d6f54686d5739952af9d6a86337c27810a2e71f95d49efade528447794e53018a e587ec4269d2ff63a25b0dc3716e63f9 5c611e85bfd0d6c50c906dfded565a0c 1.0.6.020 0 1567438620 1567440090 1 1 26 1133366485
Sep  2 16:01:30 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:01:30 2019: PCS: recv request, app_id: app_1w5B6O4R2o1k881k12177, ip: 27.18.224.156, os: 0, device_id: 88743977340299423bdb63baad439b3e
Sep  2 16:01:30 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:01:30 2019: PCS: allocate AN, app_id: app_1w5B6O4R2o1k881k12177, ip: 27.18.224.156, AN: an_1z566D44161870620, region: cn-east
Sep  2 16:01:30 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:01:30 2019: PCS: end client request 339841
Sep  2 16:01:51 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:01:51 2019: PCS: recv request 339862 from ip: 27.102.113.76:55544, public_ip: 117.136.11.24
"pcs1-20190903.log" 50466L, 9818998C

如果截取 日志文件关键字

需求:需要取出日志中的如下四个字段

  • 时间
  • app_id
  • ip
  • device_id
Sep  2 16:00:18 cc-prd-3-tk-13-pcs1 pcs[16118]: Mon Sep  2 16:00:18 2019: PCS: recv request, app_id: app_1w5B6O4R2o1k881k12177, ip: 223.104.145.136, os: 0, device_id: 02c3864050502d43dc514905133bcc9c

Gork 日志截取

编写正则

%{DATA:log_date} %{TIME:log_localtime} %{JAVAFILE:name_file} %{WORD:workd}\[%{WORD:ls}\]\: %{DATA:log_date2} %{TIME:log_localtime2} %{WORD:year_tmp}\: %{WORD:name_2}\: %{WORD:} %{WORD:}\, %{JAVAFILE:}\: %{JAVAFILE:app_id}\, %{WORD}\: %{IP:ip}\, %{WORD:}\: %{INT}\, %{WORD}\: %{USERNAME:device_id}

测试地址:grokdebug.herokuapp.com

同样也可以使用最新版的kibana 里面的Debug模块



来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/31509949/viewspace-3149278/ ,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论
管理员

注册时间:2018-03-30

  • 博文量
    2383
  • 访问量
    984187