源码安装ELK-5.6.10版本

it2022-05-05 90

一、介绍

二、安装JDK

三、安装Elasticsearch

四、安装Kibana

五、安装Nginx

六、安装Logstash

七、安装Logstash-forwarder

八、测试

系统环境：CentOS Linux release 7.4.1708 (Core)

软件版本：

elasticsearch-5.6.10

kibana-5.6.10

logstash-5.6.10

当前问题状况

开发人员不能登录线上服务器查看详细日志。各个系统都有日志，日志数据分散难以查找。日志数据量大，查询速度慢，或者数据不够实时。

一、介绍

1、组成

ELK由Elasticsearch、Logstash和Kibana三部分组件组成；Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用kibana 是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

2、四大组件Logstash: logstash server端用来搜集日志；Elasticsearch: 存储各类日志；Kibana: web化接口用作查寻和可视化日志；Logstash Forwarder: logstash client端用来通过lumberjack 网络协议发送日志到logstash server；

3、工作流程

在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。

二、安装JDK

配置阿里源：wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo yum clean all yum makecache Logstash的运行依赖于Java运行环境，Elasticsearch 要求至少 Java 7。 [root@controller ~]# yum install java-1.8.0-openjdk -y [root@controller ~]# java -version openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

三、安装Elasticsearch

官方下载地址：https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-10

#创建用户elasticsearch-5.6.10 [root@elk-node1 local]# useradd elasticsearch #解压 [root@elk-node1 application]# tar -xf elasticsearch-5.6.10.tar.gz -C /usr/local/ #创建软链接 [root@elk-node1 local]# ln -s elasticsearch-5.6.10/ elasticsearch #授权 [root@elk-node1 local]# chown -R elasticsearch elasticsearch-5.6.10/ [root@elk-node1 local]# chown -R elasticsearch elasticsearch #修改配置文件 [root@elk-node1 local]# cd elasticsearch/config/ [root@elk-node1 config]# pwd /usr/local/elasticsearch/config [root@elk-node1 config]# grep -Ev "^#|^$" elasticsearch.yml cluster.name: pcidata-elk node.name: elk-node1 bootstrap.memory_lock: true network.host: 0.0.0.0 http.port: 9200 http.cors.enabled: true http.cors.allow-origin: "*" #修改打开文件句柄数，添加limits.conf内容 [root@elk-node1 config]# ulimit -SHn 65536 [root@elk-node1 config]# tail /etc/security/limits.conf elasticsearch hard nofile 131072 elasticsearch soft nproc 2048 elasticsearch hard nproc 4096 elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited #切换到elasticsearch启动服务 [root@elk-node1 config]# su - elasticsearch Last login: Mon Aug 6 18:42:34 CST 2018 on pts/2 [elasticsearch@elk-node1 ~]$ ll total 0 [elasticsearch@elk-node1 ~]$ cd /usr/local/elasticsearch/bin/ [elasticsearch@elk-node1 bin]$ ./elasticsearch -d #报错则根据日志信息填坑 [elasticsearch@elk-node1 logs]$ pwd /usr/local/elasticsearch/logs [elasticsearch@elk-node1 logs]$ tail pcidata-elk.log [elasticsearch@elk-node1 logs]$ ss -lntp|grep 9200 LISTEN 0 128 :::9200 :::* users:(("java",pid=4189,fd=170)) #输出以下信息说明安装ok [elasticsearch@elk-node1 logs]$ curl 'http://localhost:9200/?pretty' { "name" : "elk-node1", "cluster_name" : "pcidata-elk", "cluster_uuid" : "GrfwFbeOQAmATCqZnvsq8Q", "version" : { "number" : "5.6.10", "build_hash" : "b727a60", "build_date" : "2018-06-06T15:48:34.860Z", "build_snapshot" : false, "lucene_version" : "6.6.1" }, "tagline" : "You Know, for Search" } 安装elasticsearch-head插件 1、安装git [root@elk-node1 config]# yum install -y git 2、下载elasticsearch-head插件源码 [root@elk-node1 config]# git clone git://github.com/mobz/elasticsearch-head.git Cloning into 'elasticsearch-head'... remote: Counting objects: 4224, done. remote: Total 4224 (delta 0), reused 0 (delta 0), pack-reused 4224 Receiving objects: 100% (4224/4224), 2.16 MiB | 542.00 KiB/s, done. Resolving deltas: 100% (2328/2328), done. 3.安装node 由于head插件本质上还是一个nodejs的工程，因此需要安装node，使用npm来安装依赖的包，npm可以理解为maven [root@elk-node1 elasticsearch]# wget https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.xz 4.解压node [root@elk-node1 elasticsearch]# mv node-v8.11.3-linux-x64.tar.xz /usr/local/application/ [root@elk-node1 application]# tar -xf node-v8.11.3-linux-x64.tar.xz [root@elk-node1 application]# ln -s node-v8.11.3-linux-x64/ node 5.配置node环境变量 [root@elk-node1 application]# vim /etc/profile export NODE_HOME=/usr/local/application/node export PATH=$NODE_HOME/bin:$PATH [root@elk-node1 application]# source /etc/profile 6.测试node是否生效 [root@elk-node1 application]# node -v v8.11.3 [root@elk-node1 application]# npm -v 5.6.0 初始化 [root@elk-node1 application]# npm init This utility will walk you through creating a package.json file. It only covers the most common items, and tries to guess sensible defaults. See `npm help json` for definitive documentation on these fields and exactly what they do. Use `npm install <pkg>` afterwards to install a package and save it as a dependency in the package.json file. Press ^C at any time to quit. package name: (application) version: (1.0.0) description: entry point: (index.js) test command: git repository: keywords: author: license: (ISC) About to write to /usr/local/application/package.json: { "name": "application", "version": "1.0.0", "description": "", "main": "index.js", "dependencies": { "grunt-cli": "^1.2.0" }, "devDependencies": {}, "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "author": "", "license": "ISC" } Is this ok? (yes) yes 7.安装grunt grunt是一个很方便的构建工具，可以进行打包压缩、测试、执行等等的工作，5.6.10里的head插件就是通过grunt启动的。因此需要安装一下 [root@elk-node1 elasticsearch-head]# pwd /usr/local/elasticsearch/elasticsearch-head [root@elk-node1 elasticsearch-head]# npm install -g grunt-cli 8.检查安装是否成功 [root@elk-node1 elasticsearch-head]# grunt -version grunt-cli v1.2.0 9.修改服务器监听地址 vim elasticsearch-head/Gruntfile.js connect: { server: { options: { port: 9100, hostname: '*', base: '.', keepalive: true } } } 增加hostname属性设置为* 10.修改head的连接地址: vim elasticsearch-head/_site/app.js this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://localhost:9200"; 把localhost修改成你es的服务器地址，如: this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://172.20.10.198:9200"; 11.运行head，在head目录中，执行npm install [root@elk-node1 elasticsearch-head]# npm install phantomjs-prebuilt@5.6.106 --ignore-scripts [root@elk-node1 elasticsearch-head]# npm install 12.启动nodejs grunt server & 13.访问target：9100 http://172.20.10.198:9100/

四、安装Kibana

官方下载地址：https://www.elastic.co/downloads/past-releases/kibana-5-6-10

#解压kibana-5.6.10 [root@elk-node1 local]# pwd /usr/local [root@elk-node1 local]# tar -xf kibana-5.6.10-linux-x86_64.tar.gz #创建软链接 [root@elk-node1 local]# ln -s kibana-5.6.10-linux-x86_64/ kibana #修改配置文件 [root@elk-node1 config]# pwd /usr/local/kibana/config [root@elk-node1 config]# grep -Ev "^#|^$" kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://172.20.10.198:9200" kibana.index: ".kibana" #启动 [root@elk-node1 config]# /usr/local/kibana/bin/kibana #输出以下信息说明安装ok [root@elk-node1 config]# curl localhost:5601 <script>var hashRoute = '/app/kibana'; var defaultRoute = '/app/kibana'; var hash = window.location.hash; if (hash.length) { window.location = hashRoute + hash; } else { window.location = defaultRoute; }</script>

五、安装Nginx

直接yum安装： yum install nginx httpd-tools -y 修改配置文件： vim /etc/nginx/nginx.conf 查看http{...}中是否有include /etc/nginx/default.d/*.conf;若无则添加添加kibana.conf文件： [root@controller ~]# cat /etc/nginx/conf.d/kibana.conf server { listen 80; server_name example.com; location / { proxy_pass http://172.20.10.198:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } ｝启动： [root@controller ~]# systemctl start nginx

六、安装Logstash

下载安装Logstash： [root@controller elk]# wget https://download.elastic.co/logstash/logstash/logstash-5.6.10.tar.gz [root@controller elk]# tar xf logstash-5.1.1.tar.gz [root@controller elk]# cd logstash-5.6.10 验证： [root@controller logstash-5.6.10]# ./bin/logstash -e 'input { stdin { } } output { stdout {} }' Settings: Default filter workers: 2 Logstash startup completed 2017-11-17T03:32:02.825Z controller 配置openssl: [root@controller tls]# vim /etc/pki/tls/openssl.cnf [root@controller tls]# echo " subjectAltName=IP: 172.20.10.198">>/etc/pki/tls/openssl.cnf 生成logstash-forwarder.crt文件： [root@controller tls]# openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt 配置logstash： [root@controller logstash-5.6.10]# pwd /usr/local/src/elk/logstash-5.6.10 [root@controller logstash-5.6.10]# mkdir conf [root@controller logstash-5.6.10]# cd conf/ [root@controller conf]# cat simple.conf input { lumberjack { port => 5043 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } } output { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } } 启动： [root@controller tls]# cd /usr/local/src/elk/logstash-5.6.10/bin/ [root@controller bin]# ./logstash -f ../conf/simple.conf

七、安装Logstash-forwarder(客户端)

安装源： [root@controller ~]# rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch [root@controller ~]# cat /etc/yum.repos.d/logstash-forwarder.repo name=logstash-forwarder repository baseurl=http://packages.elastic.co/logstashforwarder/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1 [root@controller ~]# yum -y install logstash-forwarder 修改配置文件：添加服务端ssl文件 mkdir -p /etc/pki/tls/certs scp user@logstash_server:/etc/pki/tls/certs/logstash_forwarder.crt /etc/pki/tls/certs/ [root@controller bin]# grep -Ev '#|^$' /etc/logstash-forwarder.conf { "network": { "servers": [ "172.20.10.198:5043" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/var/log/syslog", "/var/log/auth.log" ], "fields": { "type": "syslog" } }, { }, { } ] } 启动： service logstash-forwarder start

效果图：

转载于:https://www.cnblogs.com/Dev0ps/p/7852156.html

专利

最新回复(0)