[logstash-forwarder + logstash + elasticsearch + kibana]------------------------------------------------------------------------------------------------------------------------------------------------摘要:logstash-forwarder搜集日志，汇总给logstash，然后输出到elasticsearch,并由kibana展现web界面.------------------------------------------------------------------------------------------------------------------------------------------------一 安装1.logstash-forwardersee and install:https://github.com/elasticsearch/logstash-forwarder(logstash-forwarder有个坑. 虽然严格讲不算是logstash-forwarder的坑. 跟证书相关的:https://github.com/elasticsearch/logstash-forwarder/issues/221 <-可以不看.下面的解决方案规避这个坑了. 下面会提到.)2.logstashsee and install: http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash3.elasticsearch3.1.下载https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz 3.2.解压到目录 elasticsearch-1.3.23.3. 测试安装是否成功$ cd elasticsearch-1.3.2/$ bin/elasticsearch$ curl -X GET http://localhost:9200/(保持elasticsearch一直运行. 下面将继续测试)4.kibana:4.1.下载https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz4.2. 解压到目录 kibana-3.1.04.3. 测试安装是否成功$ cd kibana-3.1.0$ vi config.js第32行修改为：elasticsearch: "http://localhost:9200",或者如果是要非本地访问,就应该这样:elasticsearch: "http://"+window.location.hostname+":9200"注意后面有逗号. 在浏览器里打开这目录里的index.html.------------------------------------------------------------------------------------------------------------------------------------------------二 .方案:client[logstash-forwarder]---|client[logstash-forwarder]---|---log-server[logstash]--->[elasticsearch]client[logstash-forwarder]---|2.1 先启动elasticsearch前面已经启动了.2.2 开启logstash先写logstash的配置文件:$ cd logstash-1.4.2$ vi test_logstash.confinput { lumberjack { # The port to listen on port => 5000 # The paths to your ssl cert and key ssl_certificate => "/home/xiaou/logstash-forwarder.crt" ssl_key => "/home/xiaou/logstash-forwarder.key" # Set this to whatever you want. type => "somelogsXXX" }}output { elasticsearch { host => localhost } # 因为logstash和elasticsearch在同一台机器上，所以这里可以用localhost stdout { codec => rubydebug }}还要产生自签证书:$ openssl req -subj '/CN=localhost/' -x509 -batch -nodes -newkey rsa:2048 -keyout /home/xiaou/logstash-forwarder.key -out /home/xiaou/logstash-forwarder.crt -days 1095(这里用“-subj '/CN=localhost/'”规避了上面提到的logstash-forwarder的坑)然后启动logstash：$ bin/logstash -f test_logstash.conf2.3 启动logstash-forwarder先写logstash-forwarder的配置文件:$ cd logstash-forwarder$ vi test_forwarder.conf{ "network": { "servers": [ "localhost:5000" ], "ssl ca": "/home/xiaou/logstash-forwarder.crt", "timeout": 5 }, "files": [ { "paths": [ "/var/log/linshi.txt", "/var/log/*.log" ], "fields": { "type": "linshiXX" } } ]}(这里配置文件的写法也是规避了前面提到的logstash-forwarder的坑:servers没用ip)启动logstash-forwarder:$ ./logstash-forwarder -config test_forwarder.conflogstash-forwarder启动后就会与logstash建立tcp连接.测试, 写日志，观察运行logstash的终端的输出:$ echo 1234 >> /var/log/linshi.txt2.4 打开kibana,展现最终汇总到elasticsearch的日志.(唯kibana不能算是服务, 它只是一个“阅读器”.)用浏览器打开kibana-3.1.0目录下的index.html,看右边倒数第五行有个链接。打开.------------------------------------------------------------------------------------------------------------------------------------------------三.深入：1. typelogstash.conf里的input { lumberjack { ... type => "this forwarder's file have no type!"这个type,是对forwarder.conf的补充:如果forwarder.conf里没有type，则这里的type就会填充日志event的type字段.ps:一条日志event是这样的:{ "message" => "xx", "@version" => "1", "@timestamp" => "2014-09-18T03:31:12.744Z", "type" => "linshi1", "file" => "/var/log/epoch/linshi.txt", "host" => "xiaou-mint", "offset" => "568"}用type来作为区分各个日志应该不错:在forwarder里这样写files: "files": [ { "paths": [ "/var/log/epoch/linshi1.txt" ], "fields": { "type": "linshi1" } }, { "paths": [ "/var/log/epoch/linshi2.txt" ], "fields": { "type": "linshi2" } } ]2.add_field添加字段 add_field => { "test_field" => "asdasd" "test_filed2" => "112233" } 尽量不要跟日志event里已有的字段冲突了，如果要这么做，需要自行测试是否会覆盖event日志的字段. 我测试了几个字段诸如type、message、file，居然表现各一,无法统一下结论.3.if表达式随时需要查文档http://logstash.net/docs/1.4.2/。。。不写了. End./*http://logstash.net/docs/1.4.2/inputs/lumberjackhttp://logstash.net/docs/1.4.2/configuration#conditionalshttp://logstash.net/docs/1.4.2/filters/mutatehttp://logstash.net/docs/1.4.2/filters/drop*/4. 最后给出两个conf的测试内容:logstash.conf :
input { lumberjack { # The port to listen on port => 5000 # The paths to your ssl cert and key ssl_certificate => "/home/xiaou/logstash-forwarder.crt" ssl_key => "/home/xiaou/logstash-forwarder.key" type => "this forwarder's file have no type!" } } filter{ if [type] == "linshi2"{ mutate{ replace => ["message","%{message}:it's linshi2"] update => ["file", "FILE_LINSHI2"] # 替换字段. } }else{ # linshi1 if "error" in [message]{ # 日志里还有“error”字符串 mutate{ add_field => {"NOTE" => "ERROR!"} # 添加字段 add_tag => "tag_error!" # 添加标签. 标签是个数组 add_tag => "tag_error2!" } }else{ # 如果来自linshi1.txt的并且没有“error”自负, 则丢弃. drop{} } } } output { elasticsearch { host => localhost } stdout { codec => rubydebug } }forwarder.conf :
{ "network": { "servers": [ "localhost:5000" ], "ssl ca": "/home/xiaou/logstash-forwarder.crt", "timeout": 5 }, "files": [ { "paths": [ "/var/log/epoch/linshi1.txt" ], "fields": { "type": "linshi1" } }, { "paths": [ "/var/log/epoch/linshi2.txt" ], "fields": { "type": "linshi2" } } ] }------------------------------------------------------------------------------------------------------------------------------------------------End.
转载于:https://www.cnblogs.com/xiaouisme/p/3977721.html
