Centos7下使用ELK（Elasticsearch + Logstash + Kibana）搭建日志集中分析平台

日志监控和分析在保障业务稳定运行时，起到了很重要的作用，不过一般情况下日志都分散在各个生产服务器，且开发人员无法登陆生产服务器，这时候就需要一个集中式的日志收集装置，对日志中的关键字进行监控，触发异常时进行报警，并且开发人员能够查看相关日志。logstash+elasticsearch+kibana3就是实现这样功能的一套系统，并且功能更强大。

Logstash：负责日志的收集，处理和储存

Elasticsearch：负责日志检索和分析

Kibana：负责日志的可视化

1、环境介绍

elkServer

IP:192.168.7.27

OS：Centos7.1

FQDN:

elkClient

IP:192.168.31.23

OS：Centos7.1

2、下载准备

官网下载最新的安装包：https://www.elastic.co/downloads（目前有些版本的包可能下载不到了，请到该地址下载——链接：/s/1gfohO2Z 密码：5s1f）

elasticsearch-1.7.3.noarch.rpm （server上安装）kibana-4.1.2-linux-x64.tar.gz （server上安装）logstash-1.5.4-1.noarch.rpm （server上安装）logstash-forwarder-0.4.0-1.x86_64.rpm （client上安装）

3、Server端安装

3.1安装jdk1.7

[root@localhost ~]# yum install java-1.7.0-openjdkLoaded plugins: fastestmirror, langpacksbase | 3.6 kB 00:00:00extras| 3.4 kB 00:00:00updates | 3.4 kB 00:00:00Loading mirror speeds from cached hostfile* base: * extras: * updates: Package 1:java-1.7.0-openjdk-1.7.0.91-2.6.2.1.el7_1.x86_64 already installed and latest versionNothing to do

3.2安装elasticsearch

[root@localhost elk]# yum localinstall elasticsearch-1.7.3.noarch.rpm(yum本地安装elasticsearch）Loaded plugins: fastestmirror, langpacksExamining elasticsearch-1.7.3.noarch.rpm: elasticsearch-1.7.3-1.noarchelasticsearch-1.7.3.noarch.rpm: does not update installed package.Nothing to do[root@localhost elk]# systemctl daemon-reload [root@localhost elk]# systemctl enable elasticsearch.service（设置开机自启动）ln -s '/usr/lib/systemd/system/elasticsearch.service' '/etc/systemd/system/multi-user.target.wants/elasticsearch.service'[root@localhost elk]# systemctl start elasticsearch.service（开启服务）[root@localhost elk]# systemctl status elasticsearch.service（查看服务状态）elasticsearch.service - ElasticsearchLoaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)Active: active (running) since Sun -11-08 11:05:09 CST; 28s agoDocs: ain PID: 15345 (java)CGroup: /system.slice/elasticsearch.service?..15345 java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+Heap...Nov 08 11:05:09 localhost.localdomain systemd[1]: Started Elasticsearch.[root@localhost elk]# rpm -qc elasticsearch/etc/elasticsearch/elasticsearch.yml/etc/elasticsearch/logging.yml/etc/init.d/elasticsearch/etc/sysconfig/elasticsearch/usr/lib/sysctl.d/elasticsearch.conf/usr/lib/systemd/system/elasticsearch.service/usr/lib/tmpfiles.d/elasticsearch.conf[root@localhost elk]# netstat -nltp（查看端口监听状况）Active Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 00 0.0.0.0:111 0.0.0.0:*LISTEN784/rpcbind tcp 00 0.0.0.0:22 0.0.0.0:*LISTEN1457/sshd tcp 00 127.0.0.1:631 0.0.0.0:*LISTEN3213/cupsdtcp 00 127.0.0.1:25 0.0.0.0:*LISTEN2656/master tcp 00 127.0.0.1:60100.0.0.0:*LISTEN14407/sshd: root@pt tcp6 00 :::111 :::*LISTEN784/rpcbindtcp600 :::9200 :::*LISTEN15345/javatcp6 00 :::9300 :::*LISTEN15345/javatcp6 00 :::22 :::*LISTEN1457/sshd tcp6 00 ::1:631 :::*LISTEN3213/cupsdtcp6 00 ::1:25 :::*LISTEN2656/master tcp6 00 ::1:6010:::*LISTEN14407/sshd: root@pt [root@localhost elk]# firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}（防火墙添加两个端口）success[root@localhost elk]# firewall-cmd --reload（重载防火墙）success[root@localhost elk]# firewall-cmd --list-all（查看防火墙开发端口）public (default, active)interfaces: ens33sources: services: dhcpv6-client sshports: 9200/tcp 9300/tcpmasquerade: noforward-ports: icmp-blocks: rich rules:

3.3安装kibana

[root@localhost elk]# tar zxf kibana-4.1.2-linux-x64.tar.gz -C /usr/local/（解压缩安装包到指定目录中）[root@localhost elk]# cd /usr/local/[root@localhost local]# lsbin etc games include kibana-4.1.2-linux-x64 lib lib64 libexec sbin share src[root@localhost local]# mv kibana-4.1.2-linux-x64/ kibana（重命名）[root@localhost local]# cd kibana/[root@localhost kibana]# lsbin config LICENSE.txt node plugins README.txt src[root@localhost kibana]# cd bin/[root@localhost bin]# ls（运行./kibana即可开启服务，但我们将其做到service）kibana kibana.bat[root@localhost bin]# cd /etc/systemd/system/[root@localhost system]# vi kibana.service（编辑kibana服务）[Service]ExecStart=/usr/local/kibana/bin/kibana[Install]WantedBy=multi-user.target[root@localhost system]# systemctl enable kibana.service（设置开机自启动）ln -s '/etc/systemd/system/kibana.service' '/etc/systemd/system/multi-user.target.wants/kibana.service'[root@localhost system]# systemctl start kibana.service（开启服务）[root@localhost system]# systemctl status kibana.service（查看服务运行状态）kibana.serviceLoaded: loaded (/etc/systemd/system/kibana.service; enabled)Active: active (running) since Sun -11-08 11:16:28 CST; 10s agoMain PID: 16131 (node)CGroup: /system.slice/kibana.service?..16131 /usr/local/kibana/bin/../node/bin/node /usr/local/kibana/bin/../src/bin/kibana.jsNov 08 11:16:28 localhost.localdomain systemd[1]: Started kibana.service.Nov 08 11:16:34 localhost.localdomain kibana[16131]: {"name":"Kibana","hostname":"localhost.localdomain","pid":16131,"level":30,"msg":"No existing kibana index found","time":"20...43Z","v":0}Nov 08 11:16:34 localhost.localdomain kibana[16131]: {"name":"Kibana","hostname":"localhost.localdomain","pid":16131,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"-11...93Z","v":0}Hint: Some lines were ellipsized, use -l to show in full.[root@localhost system]# netstat -nltp（查看端口监听状态）Active Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp00 0.0.0.0:5601 0.0.0.0:*LISTEN16131/nodetcp 00 0.0.0.0:111 0.0.0.0:*LISTEN784/rpcbind tcp 00 0.0.0.0:22 0.0.0.0:*LISTEN1457/sshd tcp 00 127.0.0.1:631 0.0.0.0:*LISTEN3213/cupsdtcp 00 127.0.0.1:25 0.0.0.0:*LISTEN2656/master tcp 00 127.0.0.1:60100.0.0.0:*LISTEN14407/sshd: root@pt tcp6 00 :::111 :::*LISTEN784/rpcbind tcp6 00 :::9200 :::*LISTEN15345/javatcp6 00 :::9300 :::*LISTEN15345/javatcp6 00 :::22 :::*LISTEN1457/sshd tcp6 00 ::1:631 :::*LISTEN3213/cupsdtcp6 00 ::1:25 :::*LISTEN2656/master tcp6 00 ::1:6010:::*LISTEN14407/sshd: root@pt [root@localhost system]# firewall-cmd --permanent --add-port=5601/tcp（防火墙开启5601端口）success[root@localhost system]# firewall-cmd --reload（重载防火墙）success[root@localhost system]# firewall-cmd --list-all（查看防火墙开放端口）public (default, active)interfaces: ens33sources: services: dhcpv6-client sshports: 9200/tcp 9300/tcp 5601/tcpmasquerade: noforward-ports: icmp-blocks: rich rules: [root@localhost system]# firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=5601 （为5601端口添加80端口的映射，这样在浏览器中就可以不用输入端口了）success[root@localhost system]# firewall-cmd --reload（重载防火墙）success[root@localhost system]# firewall-cmd --list-all（查看防火墙开放端口）public (default, active)interfaces: ens33sources: services: dhcpv6-client sshports: 9200/tcp 9300/tcp 5601/tcpmasquerade: noforward-ports: port=80:proto=tcp:toport=5601:toaddr=icmp-blocks: rich rules:

3.4安装logstash

[root@localhost system]# cd /home/elk/[root@localhost elk]# lselasticsearch-1.7.3.noarch.rpm kibana-4.1.2-linux-x64.tar.gz logstash-1.5.4-1.noarch.rpm logstash-forwarder-0.4.0-1.x86_64.rpm[root@localhost elk]# yum localinstall logstash-1.5.4-1.noarch.rpm（yum本地安装logstash）Loaded plugins: fastestmirror, langpacksExamining logstash-1.5.4-1.noarch.rpm: 1:logstash-1.5.4-1.noarchMarking logstash-1.5.4-1.noarch.rpm to be installedResolving Dependencies--> Running transaction check---> Package logstash.noarch 1:1.5.4-1 will be installed--> Finished Dependency Resolutionbase/7/x86_64 | 3.6 kB 00:00:00extras/7/x86_64 | 3.4 kB 00:00:00extras/7/x86_64/primary_db | 116 kB 00:00:00updates/7/x86_64 | 3.4 kB 00:00:00updates/7/x86_64/primary_db| 4.7 MB 00:00:03Dependencies Resolved===============================================================================================================================================================================================PackageArchVersion Repository Size===============================================================================================================================================================================================Installing:logstash noarch 1:1.5.4-1/logstash-1.5.4-1.noarch 136 MTransaction Summary===============================================================================================================================================================================================Install 1 PackageTotal size: 136 MInstalled size: 136 MIs this ok [y/d/N]: yDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transactionInstalling : 1:logstash-1.5.4-1.noarch 1/1 Verifying : 1:logstash-1.5.4-1.noarch 1/1 Installed:logstash.noarch 1:1.5.4-1 Complete![root@localhost tls]# hostname -f（查看当前FQDN，FQDN设置参见/zhenyuyaodidiao/p/4947930.html）[root@localhost ~]# cd /etc/pki/tls/（进入到/etc/pki/tls/文件夹）[root@localhost tls]# lscert.pem certs misc f private（以下生成openssl key用于客户端上传日志文件用，在客户端配置时会用到）[root@localhost tls]# openssl req -subj '/CN=/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crtGenerating a 2048 bit RSA private key..............+++.............+++writing new private key to 'private/logstash-forwarder.key'-----[root@localhost tls]# lscert.pem certs misc f private[root@localhost tls]# cd private/[root@localhost private]# lltotal 4-rw-r--r--. 1 root root 1704 Nov 8 17:20 logstash-forwarder.key[root@localhost private]# cd ../certs/[root@localhost certs]# lltotal 16lrwxrwxrwx. 1 root root 49 Apr 14 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pemlrwxrwxrwx. 1 root root 55 Apr 14 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt-rw-r--r--. 1 root root 1107 Nov 8 17:20 logstash-forwarder.crt-rwxr-xr-x. 1 root root 610 Mar 24 make-dummy-cert-rw-r--r--. 1 root root 2388 Mar 24 Makefile-rwxr-xr-x. 1 root root 829 Mar 24 renew-dummy-cert[root@localhost ~]# cd /etc/logstash/conf.d/[root@localhost conf.d]# vi 01-logstash-initial.conf（编辑logstash配置文件）input {lumberjack {port => 5000type => "logs"ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"}}filter {if [type] == "syslog" {grok {match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }add_field => [ "received_at", "%{@timestamp}" ]add_field => [ "received_from", "%{host}" ]}syslog_pri { }date {match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]}}}output {elasticsearch { host => localhost }stdout { codec => rubydebug }}[root@localhost conf.d]# systemctl enable logstash（设置开机自启动）logstash.service is not a native service, redirecting to /sbin/chkconfig.Executing /sbin/chkconfig logstash onThe unit files have no [Install] section. They are not meant to be enabledusing systemctl.Possible reasons for having this kind of units are:1) A unit may be statically enabled by being symlinked from another unit's.wants/ or .requires/ directory.2) A unit's purpose may be to act as a helper for some other unit which hasa requirement dependency on it.3) A unit may be started when needed via activation (socket, path, timer,D-Bus, udev, scripted systemctl call, ...).[root@localhost conf.d]# systemctl start logstash.service（开启logstash服务）[root@localhost conf.d]# systemctl status logstash.service（查看服务运行状态）logstash.service - LSB: Starts Logstash as a daemon.Loaded: loaded (/etc/rc.d/init.d/logstash)Active: active (running) since Sun -11-08 17:28:34 CST; 14s agoProcess: 20799 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)CGroup: /system.slice/logstash.service?..20805 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib...Nov 08 17:28:34 elk logstash[20799]: logstash started.Nov 08 17:28:34 elk systemd[1]: Started LSB: Starts Logstash as a daemon..[root@localhost conf.d]# netstat -nltp（查看端口占用）Active Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 00 0.0.0.0:5601 0.0.0.0:*LISTEN16131/nodetcp 00 0.0.0.0:111 0.0.0.0:*LISTEN784/rpcbind tcp 00 0.0.0.0:22 0.0.0.0:*LISTEN1457/sshd tcp 00 127.0.0.1:631 0.0.0.0:*LISTEN3213/cupsdtcp 00 127.0.0.1:25 0.0.0.0:*LISTEN2656/master tcp 00 127.0.0.1:60100.0.0.0:*LISTEN14407/sshd: root@pt tcp 00 127.0.0.1:60120.0.0.0:*LISTEN17715/sshd: root@pttcp600 :::5000 :::*LISTEN20805/javatcp6 00 :::111 :::*LISTEN784/rpcbind tcp6 00 :::9200 :::*LISTEN15345/javatcp6 00 :::9300 :::*LISTEN15345/javatcp6 00 :::9301 :::*LISTEN20805/javatcp6 00 :::22 :::*LISTEN1457/sshd tcp6 00 ::1:631 :::*LISTEN3213/cupsdtcp6 00 ::1:25 :::*LISTEN2656/master tcp6 00 ::1:6010:::*LISTEN14407/sshd: root@pt tcp6 00 ::1:6012:::*LISTEN17715/sshd: root@pt [root@localhost conf.d]# cd /var/log/logstash/[root@localhost logstash]# ls（日志文件）logstash.err logstash.log logstash.stdout[root@localhost logstash]# firewall-cmd --permanent --add-port=5000/tcp （防火墙开放5000端口）success[root@localhost logstash]# firewall-cmd --reload（重载防火墙）success[root@localhost logstash]# firewall-cmd --list-all（查看端口开放情况）public (default, active)interfaces: ens33sources: services: dhcpv6-client sshports: 9200/tcp 9300/tcp 5000/tcp 5601/tcpmasquerade: noforward-ports: port=80:proto=tcp:toport=5601:toaddr=icmp-blocks: rich rules:

4、Client端安装

[root@localhost elk]# vi /etc/hosts（编辑hosts文件）127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.7.27 [root@localhost elk]# service network restartRestarting network (via systemctl): [ OK ][root@localhost elk]# ping（测试连接）PING (192.168.7.27) 56(84) bytes of data.64 bytes from (192.168.7.27): icmp_seq=1 ttl=63 time=0.754 ms64 bytes from (192.168.7.27): icmp_seq=2 ttl=63 time=0.477 ms^C--- ping statistics ---2 packets transmitted, 2 received, 0% packet loss, time 1000msrtt min/avg/max/mdev = 0.477/0.615/0.754/0.140 ms[root@localhost laizy]# mkdir elk[root@localhost laizy]# cd elk/[root@localhost elk]# ls[root@localhost elk]# scp root@192.168.7.27:/home/elk/logstash-forwarder-0.4.0-1.x86_64.rpm .（拷贝logstash-forwarder到本地）The authenticity of host '192.168.7.27 (192.168.7.27)' can't be established.ECDSA key fingerprint is 49:b9:53:89:55:f2:93:87:9b:81:bb:23:a5:24:f1:f9.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '192.168.7.27' (ECDSA) to the list of known hosts.root@192.168.7.27's password: logstash-forwarder-0.4.0-1.x86_64.rpm100% 1692KB 1.7MB/s 00:00 [root@localhost elk]# lslogstash-forwarder-0.4.0-1.x86_64.rpm[root@localhost elk]# scp root@192.168.7.27:/etc/pki/tls/certs/logstash-forwarder.crt .（拷贝Server端的key到本地）root@192.168.7.27's password: logstash-forwarder.crt100% 11071.1KB/s 00:00 [root@localhost elk]# lltotal 1700-rw-r--r--. 1 root root 1732758 Nov 8 17:36 logstash-forwarder-0.4.0-1.x86_64.rpm-rw-r--r--. 1 root root 1107 Nov 8 17:37 logstash-forwarder.crt[root@localhost elk]# cp logstash-forwarder.crt /etc/pki/tls/certs/（将key拷贝到/etc/pki/tls/certs/下）[root@localhost elk]# cd /etc/pki/tls/certs/[root@localhost certs]# lsca-bundle.crt ca-bundle.trust.crt logstash-forwarder.crt make-dummy-cert Makefile renew-dummy-cert[root@localhost certs]# cd /home/laizy/elk/[root@localhost elk]# lslogstash-forwarder-0.4.0-1.x86_64.rpm logstash-forwarder.crt[root@localhost elk]# yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm（yum本地安装logstash-forwarder）Loaded plugins: fastestmirror, langpacksExamining logstash-forwarder-0.4.0-1.x86_64.rpm: logstash-forwarder-0.4.0-1.x86_64Marking logstash-forwarder-0.4.0-1.x86_64.rpm to be installedResolving Dependencies--> Running transaction check---> Package logstash-forwarder.x86_64 0:0.4.0-1 will be installed--> Finished Dependency Resolutionbase/7/x86_64| 3.6 kB 00:00:00extras/7/x86_64 | 3.4 kB 00:00:00updates/7/x86_64 | 3.4 kB 00:00:00Dependencies Resolved===============================================================================================================================================================================================PackageArch Version Repository Size===============================================================================================================================================================================================Installing:logstash-forwarder x86_64 0.4.0-1 /logstash-forwarder-0.4.0-1.x86_64 5.7 MTransaction Summary===============================================================================================================================================================================================Install 1 PackageTotal size: 5.7 MInstalled size: 5.7 MIs this ok [y/d/N]: yDownloading packages:Running transaction checkRunning transaction testTransaction test succeededRunning transactionInstalling : logstash-forwarder-0.4.0-1.x86_64 1/1 Logs for logstash-forwarder will be in /var/log/logstash-forwarder/Verifying : logstash-forwarder-0.4.0-1.x86_64 1/1 Installed:logstash-forwarder.x86_64 0:0.4.0-1 Complete![root@localhost elk]# systemctl enable logstash-forwarder（设置开机自启动）logstash-forwarder.service is not a native service, redirecting to /sbin/chkconfig.Executing /sbin/chkconfig logstash-forwarder onThe unit files have no [Install] section. They are not meant to be enabledusing systemctl.Possible reasons for having this kind of units are:1) A unit may be statically enabled by being symlinked from another unit's.wants/ or .requires/ directory.2) A unit's purpose may be to act as a helper for some other unit which hasa requirement dependency on it.3) A unit may be started when needed via activation (socket, path, timer,D-Bus, udev, scripted systemctl call, ...).[root@localhost elk]# systemctl start logstash-forwarder.service（开启服务）[root@localhost elk]# cd /var/log/logstash-forwarder/（日志目录）[root@localhost logstash-forwarder]# lslogstash-forwarder.err logstash-forwarder.log[root@localhost elk]# vi /etc/logstash-forwarder.conf（编辑配置文件）{"network": {"servers": [ ":5000" ],"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt","timeout": 15},"files": [{"paths": ["/var/log/messages","/var/log/secure"],"fields": { "type": "syslog" }}]}[root@localhost elk]# systemctl restart logstash-forwarder.service（重启服务）[root@localhost elk]# systemctl status logstash-forwarder.service（查看服务运行状态）logstash-forwarder.service - LSB: no description givenLoaded: loaded (/etc/rc.d/init.d/logstash-forwarder)Active: active (running) since Sun -11-08 18:30:51 CST; 18s agoProcess: 10788 ExecStop=/etc/rc.d/init.d/logstash-forwarder stop (code=exited, status=0/SUCCESS)Process: 10794 ExecStart=/etc/rc.d/init.d/logstash-forwarder start (code=exited, status=0/SUCCESS)CGroup: /system.slice/logstash-forwarder.service?..10798 /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder.confNov 08 18:30:51 localhost.localdomain systemd[1]: Starting LSB: no description given...Nov 08 18:30:51 localhost.localdomain /etc/init.d/logstash-forwarder[10799]: logstash-forwarder startedNov 08 18:30:51 localhost.localdomain logstash-forwarder[10794]: logstash-forwarder startedNov 08 18:30:51 localhost.localdomain systemd[1]: Started LSB: no description given.

5、界面验证

首先在client中手动增加一条日志：

[root@localhost elk]# logger zhenyuLogtest

界面登录 http://192.168.7.27/，做如下操作

从图中可以看到，手动添加的日志已经在界面中被搜索到了。

本文主要参考了国外一个搭建ELK的视频，操作的很详细，附上视频的下载链接，仅供参考。

链接：/s/1jGuBWCQ 密码：h0pq