ITPub博客

首页 > 大数据 > 数据分析 > linux下Logstash对系统日志,web日志集中式管理

linux下Logstash对系统日志,web日志集中式管理

数据分析 作者:liujian66686 时间:2013-09-13 10:41:00 0 删除 编辑
Logstash_elasticsearch_redis是对linux系统日志,邮件日志,web日志等一切文本格式日志提供集中式管理的一种快速管理方案,方便日志的分析查看。为了操作系统、业务系统错误日志集中式管理,linux系统实
战就这个方案的安装做了一次详细的实战操作,其中Kibana是web查询管理界面,简洁高效。经测试,Elasticsearch对文本日志的索引几乎
是实时的。redis这里扮演了一个日志队列中间件,对logstash
agent过滤出来的日志加入redis队列中。也是非常高效,好了,下面引用了英文说明Getting Started Centralized Setup with Event Parsing    Download and run Elasticsearch    Download and run Redis    Download and run Logstash    web interface Kibana    https://logstash.objects.dreamhost.com/release/logstash-1.1.10-flatjar.jar    https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.0.RC2.tar.gz    http://redis.googlecode.com/files/redis-2.6.12.tar.gz    https://nodeload.github.com/rashidkpc/Kibana/tar.gz/v0.2.0   现在是使用Elasticsearch+Redis+Logstash,   现在用redis代替之前是使用  AMQP 的 RabbitMQ方式,理由是Redis has no external dependencies and has a much simpler configuration in Logstash.一:安装jdk 略二:   安装redis   tar xvf redis-2.6.12.tar.gz   cd redis-2.6.12   make   make install   之后出现   mkdir -p /usr/local/bin    INSTALL install    INSTALL install    INSTALL install    INSTALL install    INSTALL installmake[1]: Leaving directory `/root/loghash/redis-2.6.12/src'redis安装在/usr/local/bin目录中# ls /usr/local/bin/ | grep redisredis-benchmarkredis-check-aofredis-check-dumpredis-cliredis-server有以上的程序,说明redis安装成功。启动/usr/local/bin/redis-server --loglevel verbose# /usr/local/bin/redis-server --loglevel verbose                _._                                                             _.-``__ ''-._                                                   _.-``    `.  `_.  ''-._           Redis 2.6.12 (00000000/0) 64 bit  .-`` .-```.  ```/    _.,_ ''-._                                    (    '      ,       .-`  | `,    )     Running in stand alone mode |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379 |    `-._   `._    /     _.-'    |     PID: 5405  `-._    `-._  `-./  _.-'    _.-'                                    |`-._`-._    `-.__.-'    _.-'_.-'|                                   |    `-._`-._        _.-'_.-'    |           http://redis.io          `-._    `-._`-.__.-'_.-'    _.-'                                    |`-._`-._    `-.__.-'    _.-'_.-'|                                   |    `-._`-._        _.-'_.-'    |                                    `-._    `-._`-.__.-'_.-'    _.-'                                         `-._    `-.__.-'    _.-'                                                 `-._        _.-'                                                         `-.__.-'                                               [5405] 19 Apr 11:23:52.165 # Server started, Redis version 2.6.12[5405] 19 Apr 11:23:52.165 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition.To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and
then reboot or run the command 'sysctl vm.overcommit_memory=1' for this
to take effect.[5405] 19 Apr 11:23:52.165 * The server is now ready to accept connections on port 6379[5405] 19 Apr 11:23:52.165 - 0 clients connected (0 slaves), 798312 bytes in use[5405] 19 Apr 11:23:57.165 - 0 clients connected (0 slaves), 798312 bytes in use针对这个WARNING overcommit_memory is set to 0,把overcommit_memory设置成1echo "vm.overcommit_memory = 1"  >> /etc/sysctl.confsysctl -predis以服务方式启动mkdir -p /etc/redis/cd redis-2.6.12scp redis.conf /etc/redis/把redis.conf修改其中的daemonize no改成daemonize yesuseradd -M redis -s /sbin/nologinredis启动脚本见附件redis.init.shchkconfig --add redischkconfig --level 35 redis onservice redis start三:安装elasticsearchtar xvf elasticsearch-0.90.0.RC2.tar.gzmv elasticsearch-0.90.0.RC2 /usr/local/elasticsearch必要时改下elasticsearch的集群名vi /usr/local/elasticsearch/config/elasticsearch.ymlcluster.name: xiutuo_elasticsearch启动elasticsearch,  加上-f 方便调试信息的输出/usr/local/elasticsearch/bin/elasticsearch -f四:安装loghash如果我们使用loghash的集中模式 centralized 的话(loghash有standalone和centralized俩种方式),就有俩种agent角色: shipper 和 indexer其中agent shipper将会从所有服务器中通过redis传输日志,而agent indexer将会接受和分析过滤日志,然后通过 elasticsearch 生成索引Since we're doing a centralized configuration, you'll have two main logstash agent roles: a shipper and an indexer.You will ship logs from all servers via Redis and have another agent
receive those messages, parse them, and index them in elasticsearch.3.1 安装loghash agent shipper,安装在需要日志分析的服务器上把logstash-1.1.10-flatjar.jar放在/usr/local/loghash里mkdir -p /usr/local/loghashscp /root/loghash/logstash-1.1.10-flatjar.jar /usr/local/loghash/在需要日志分析的服务器上,配置开启动agent shipper配置agent shipper: 文件名可以为任意,文件可以放在任意位置(有权限读入即可),我这里为了方面区分,直接使用shipper.conf,并放在/etc/下cat > /etc/shipper.conf < "stdin-type"  }}output {  stdout { debug => true debug_format => "json"}  redis { host => "127.0.0.1" data_type => "list" key => "logstash" }}EOF启动agent shipper此时,你在上面的那个命令下面随便输入,如回车,在/usr/local/bin/redis-server --loglevel verbose 命令窗口中就会显示[5463] 19 Apr 11:52:31.250 - DB 0: 1 keys (0 volatile) in 4 slots HT.[5463] 19 Apr 11:52:31.250 - 1 clients connected (0 slaves), 830040 bytes in use[5463] 19 Apr 11:52:36.250 - DB 0: 1 keys (0 volatile) in 4 slots HT.[5463] 19 Apr 11:52:36.250 - 1 clients connected (0 slaves), 830280 bytes in use引用原文This will take anything you type into this console and display it on the
console. Additionally it will save events to Redis in a list named
after the key value you provided.通过redis客户端redis-cli,输入llen logstash和lpop logstash 就可以看到如下信息# redis-cliredis 127.0.0.1:6379> llen logstash(integer) 55redis 127.0.0.1:6379> lpop logstash"{"@source":"stdin://el-test1/","@tags":[],"@fields":{},"@timestamp":"2013-04-19T03:49:52.320Z","@source_host":"el-test1","@source_path":"/","@message":"","@type":"stdin-type"}"redis 127.0.0.1:6379>到次为止,loghash agent shiper就配置成功了3.2 安装loghash agent indexer,安装在要分析的日志服务器上这个agent indexer 在redis上分析和索引服务器日志This agent will parse and index your logs as they come in over Redis.
Here's a sample config based on the previous section. Save this as
indexer.conf更agent shiper配置一样,我们把loghash agent indexer配置文件命名成indexer.conf,放在/etc下cat /etc/indexer.confinput {  redis {    host => "127.0.0.1"    type => "redis-input"    # these settings should match the output of the agent    data_type => "list"    key => "logstash"    # We use json_event here since the sender is a logstash agent    format => "json_event"  }}output {  stdout { debug => true debug_format => "json"}  elasticsearch {    host => "127.0.0.1"  }}启动java -jar  /usr/local/loghash/logstash-1.1.10-flatjar.jar agent -f /etc/indexer.conf为了检验agent indexer开启后Elasticsearch是否有在写日志呢。请到 /usr/local/elasticsearch/bin/elasticsearch -f  这个启动(debug模式)下,如果看到类似的[2013-04-19 12:50:18,864][INFO ][cluster.service          ] [Shadrac]
new_master [Shadrac][UOETxFzaSuyEsIS_tul08w][inet[/192.168.0.8:9301]],
reason: zen-disco-join (elected_as_master)这样的信息就说明配置agent indexer 和 elasticsearch都成功了。好了,到此步骤为止,Elasticsearch+redis+loghash部分的前期工作都做完了,剩下就是loghash的logstash web interface 即web界面的配置了这里loghash使用kibana的web UItar xvf Kibana-0.2.0.tar.gzmv Kibana-0.2.0 /usr/local/kibanavi /usr/local/kibana/KibanaConfig.rb必要时候设置 Elasticsearch = "localhost:9200"    KibanaHost = '127.0.0.1' 改成  Elasticsearch = "localhost:9200"    KibanaHost = '0.0.0.0' 安装ruby相关组件  yum -y install gcc+ gcc-c++ ruby ruby-devel rubygems    gem install bundler  bundle install启动kibana (cole模式即可debug模式)ruby kibana.rb以服务(deamon)方式启动脚本cat /etc/init.d/kibana    #!/bin/sh    #    # kibana    Web interface to Logstash.    #    # chkconfig:   - 85 15    # description: Kibana is a web interface to Logstash.    ### BEGIN INIT INFO    # Provides: kibana    # Required-Start: $all    # Required-Stop: $all    # Default-Start:    # Default-Stop: 0 1 6    # Short-Description: Kibana    # Description: Kibana is a web interface to Logstash.    ### END INIT INFO    # Source function library.    . /etc/rc.d/init.d/functions    prog="kibana"    [ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog    if [ "x$KIBANA_HOME" == "x" ]; then        KIBANA_HOME="/usr/local/kibana"    fi    export GEM_PATH="$KIBANA_HOME/vendor/bundle/ruby/1.*"    bin="$KIBANA_HOME/kibana-daemon.rb"    exec="/usr/bin/env ruby $bin"    config="$KIBANA_HOME/KibanaConfig.rb"    lockfile=/var/lock/subsys/$prog    start() {        [ -e $bin ] || exit 5        [ -f $config ] || exit 6        echo -n $"Starting $prog: "        cd $KIBANA_HOME        daemon $exec start &        retval=$?        echo        [ $retval -eq 0 ] && touch $lockfile        return $retval    }    stop() {        echo -n $"Stopping $prog: "        cd $KIBANA_HOME        daemon $exec stop        retval=$?        echo        [ $retval -eq 0 ] && rm -f $lockfile && rm -rf $KIBANA_HOME/tmp/kibana.rb.pid        return $retval    }    status() {        cd $KIBANA_HOME        $exec status    }    restart() {        stop        start    }    case "$1" in        start)            $1            ;;        stop)            $1            ;;        restart)            $1            ;;        status)            $1            ;;        *)            echo $"Usage: $0 {start|stop|status|restart}"            exit 2    esac    exit $?========================================================  出现以下错误An error occurred while installing eventmachine (1.0.0), and Bundler cannot continue.Make sure that `gem install eventmachine -v '1.0.0'` succeeds before bundling.请安装ruby-devel开启内置的UI(这个操作一定要废除。否则出现),java -jar  /usr/local/loghash/logstash-1.1.10-flatjar.jar web --backend elasticsearch://127.0.0.1/   (废除他吧)出现以下错误Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (IOError) no !/ in spec        at org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1046)        at RUBY.(root)(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/openssl.rb:20)        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1027)        at
Kernel.require(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:36)        at Kernel.require(file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/logstash/JRUBY-6970.rb:13)        at RUBY.(root)(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/ftw/connection.rb:1)        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1027)        at
Kernel.require(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:36)        at Kernel.require(file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/logstash/JRUBY-6970.rb:13)        at RUBY.(root)(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/ftw/connection.rb:8)        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1027)        at RUBY.(root)(file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/ftw/agent.rb:1)        at RUBY.(root)(file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/ftw/agent.rb:3)        at
RUBY.(root)(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:1)        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1027)        at
RUBY.require(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:36)        at RUBY.require(file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/logstash/JRUBY-6970.rb:13)        at RUBY.(root)(file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/ftw.rb:1)        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1027)        at
RUBY.(root)(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:1)        at
RUBY.require(jar:file:/usr/local/loghash/logstash-1.1.10-flatjar.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:36)        at logstash.runner.run(logstash/runner.rb:104)        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:249)        at logstash.runner.run(logstash/runner.rb:199)        at logstash.runner.main(logstash/runner.rb:74)        at logstash.runner.(root)(logstash/runner.rb:224)出现这个错误,请安装UI: Kibana安装Kibanayum -y install ruby rubygems   ruby-develgem install bundlergem install daemonsgem install sinatragem install tzinfogem install fastercsvbundle install配置KibanaKibanaHost = '127.0.0.1' 改成KibanaHost = '0.0.0.0'Elasticsearch = "localhost:9200"改成Elasticsearch = "elasticsearch所在主机IP:9200"错误 bundle installBundler::GemfileNotFoundfind/ -name "Gemfile"/usr/lib/ruby/gems/1.8/gems/bundler-1.3.5/lib/bundler/templates/Gemfilecd /usr/lib/ruby/gems/1.8/gems/bundler-1.3.5/lib/bundler/templates/# bundle install        The Gemfile specifies no dependenciesResolving dependencies...Your bundle is complete!Use `bundle show [gemname]` to see where a bundled gem is installed.[root@xiutuo ~]# service kibana startStarting kibana:[root@xiutuo ~]#
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
`gem_original_require': no such file to load -- daemons (LoadError)        from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'    解决    gem install daemons/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- sinatra (LoadError)        from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'        from kibana.rb:2        解决    gem install sinatramkmf.rb can't find header files for ruby at /usr/lib/ruby/ruby.h解决yum -y install ruby-devel好了,linux系统实战稍后会放出具体优化后的logstash对系统日志配置文件,邮件系统日志过滤,tomcat日志过滤配置文件,apache日
志过滤配置文件等直接以文件附件的方式放在相关文章后面,为对该方案logstash+Elasticsearch+redis+kibana不是了解的
用户提供直接帮助,敬请期待。

来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/24135624/viewspace-1110984/ ,如需转载,请注明出处,否则将追究法律责任。

上一篇: 没有了~
下一篇: 没有了~
请登录后发表评论 登录
全部评论

注册时间:2010-06-22