一、Elasticsearch+kibana部署server
注：此文档为傻瓜式安装，以避过所有坑，简单安装方便使用，如遇以外问题请度娘
环境部署&&版本需求
CentOS7
Elasticsearch-7.30
kibana-7.30
logstash-7.30
服务器需求两台
200.200.100.51 node1
200.200.100.52 node2
200.200.100.53 node3
1.关闭防火墙&&selinux
systemctl stop firewalldsystemctl disable firewalldsed -i 's/enforcing/disabled/g' /etc/sysconfig/selinux
2.内核参数优化
echo '* hard nofile 65536* soft nofile 65536* soft nproc 65536* hard nproc 65536'>>/etc/security/limits.conf
echo 'vm.max_map_count = 262144net.core.somaxconn=65535net.ipv4.ip_forward = 1'>>/etc/sysctl.conf
sysctl -p
3.时间同步
yum -y install ntpsystemctl enable ntpdsystemctl start ntpdntpdate -u cn.hwclock --systohctimedatectl set-timezone Asia/Shanghai
4.安装必备软件
yum install wget vim lsof net-tools lrzsz net-tools curl -y
5.配置JDK环境
tar -zxf jdk-11.0.4_linux-x64_bin.tar.gz mv jdk-11.0.4 /usr/local/jdkecho 'export JAVA_HOME=/usr/local/jdkexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$JAVA_HOME/bin:$PATH'>>/etc/profilesource /etc/profilejava -version
6.安装配置elasticserach
tar zxvf elasticsearch-7.3.0-linux-x86_64.gz mv elasticsearch-7.3.0 /usr/local/elasticsearchmkdir -p /data/{es-data,es-logs}
修改elasticsearch配置文件vim /usr/local/elasticsearch/config/elasticsearch.yml 修改后：grep -Ev "^$|#" /usr/local/elasticsearch/config/elasticsearch.yml
Node1
Node2
Node3
7.配置权限启动elasticsearch
创建用户
useradd efkchown -R efk:efk /usr/local/jdkchown -R efk:efk /usr/local/elasticsearchchown -R efk:efk /datasu - efk/usr/local/elasticsearch/bin/elasticsearch -d
8.配置启动kibana
tar zxf kibana-7.3.0-linux-x86_64.tar.gz mv kibana-7.3.0-linux-x86_64 /usr/local/kibanavim /usr/local/kibana/config/kibana.yml grep -Ev "^$|#" /usr/local/kibana/config/kibana.yml server.port: 5601server.host: "200.200.100.51"elasticsearch.host: ["http:/200.200.100.51:9200"]chown -R efk:efk /usr/local/kibanasu - efk/usr/local/kibana/bin/kibana &
访问kibana：http://ip:5601
9.配置elasticsearch-head插件
下载安装包wget /dist/v12.16.3/node-v12.16.3-linux-x64.tar.xzxz -d node-v12.16.3-linux-x64.tar.xz tar xf node-v12.16.3-linux-x64.tar -C /usr/local/mv /usr/local/node-v12.16.3-linux-x64/ /usr/local/nodeecho "export PATH=$PATH:/usr/local/node/bin" >>/etc/profile. /etc/profile && source /etc/profilenode -vyum install git bzip2 -y git clone /mobz/elasticsearch-head.gitmv elasticsearch-head /usr/local/cd /usr/local/elasticsearch-head/
```可不需要```安装 grunt-clinpm install -g grunt-cli安装 gruntelasticsearch-head 下载完成后，进入 elasticsearch-head 文件夹，执行命令：npm install grunt --save
安装依赖的 npm 包
npm install
如果出出现错误Error: Command failed: tar jxf /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2需要安装bzip2软件包
如果报错为此错误：因为fsevents是Mac系统所需的软件，用下面命令跳过此软件安装
npm install --unsafe-perm
vim /usr/local/elasticsearch-head/Gruntfile.js
vim /usr/local/elasticsearch-head/_site/app.js
将localhost 改成elasticsearch cluster的地址
chown -R efk:efk /usr/local/elasticsearch-head/su - efknpm run start & #如果无法启动的话或者/usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server &
访问：http://200.200.100.51:9100
二、 logstash部署
logstash安装的Nginx服务器上，并非es服务器
1.解压安装logstash
tar xf logstash-7.3.0.tar.gz -C /usr/local/mv /usr/local/logstash-7.3.0 /usr/local/logstash
一般情况下，我们可以不配置logstash直接就可以启动，logstash下有一个叫logstash.yml的文件，里面可以对logstash做一些简单的优化
vim /usr/local/logstash/config/logstash.ymlconfig.reload.automatic: true #开启配置文件自动加载config.reload.interval: 10 #定义配置文件重载时间周期
可以添加也可以不添加！
2.创建配置文件
input {beats {port => 5044}}output {stdout {codec => rubydebug}if [log_source] == 'weblogic_yun' {elasticsearch {hosts => ["200.200.100.51:9200","200.200.100.52:9200","200.200.100.53:9200"]index => "weblogic_yun-%{+YYYY.MM.dd}"}}if [log_source] == 'weblogic_jl' {elasticsearch {hosts => ["200.200.100.51:9200","200.200.100.52:9200","200.200.100.53:9200"]index => "weblogic_jl-%{+YYYY.MM.dd}"}}if [log_source] == 'message' {elasticsearch {hosts => ["200.200.100.51:9200","200.200.100.52:9200","200.200.100.53:9200"]index => "message-%{+YYYY.MM.dd}"}}}
三、filebeat安装
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.0-linux-x86_64.tar.gz
tar -zvxf filebeat-7.3.0-linux-x86_64.tar.gz -C /usr/local/
#=========================== Filebeat inputs =============================filebeat.inputs:- type: logenabled: truepaths:- /var/log/messagefields:log_source: messagefields_under_root: true#============================= Filebeat modules ===============================filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: false#==================== Elasticsearch template setting ==========================setup.template.settings:index.number_of_shards: 1#============================== Kibana =====================================setup.kibana:host: "200.200.100.51:5601"#----------------------------- Logstash output --------------------------------output.logstash:hosts: ["200.200.100.51:5044"]#================================ Processors =====================================processors:- add_host_metadata: ~- add_cloud_metadata: ~
五、安装并配置Nginx收集日志
安装Nginx
wget /download/nginx-1.10.3.tar.gz
yum install -y gcc glibc gcc-c++ prce-devel openssl-devel pcre-devel
useradd -s /sbin/nologin www -M
tar xf nginx-1.10.3.tar.gz && cd nginx-1.10.3
./configure --prefix=/usr/local/nginx-1.10.3 --user=www --group=www --with-http_ssl_module --with-http_stub_status_module
make && make install
ln -s /usr/local/nginx-1.10.3 /usr/local/nginx
手动启动
/usr/local/nginx/sbin/nginx
设置开机启动
echo “/usr/local/nginx/sbin/nginx” >>/etc/rc.local
查看服务器是否启动
netstat -lntp|grep nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 7058/nginx: master
需要配置在我们要收集的服务器上
vim /usr/local/nginx/conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - remoteuser[remote_user [remoteuser[time_local] “KaTeX parse error: Expected 'EOF', got '#' at position 16: request" ' #̲ …status bodybytessent"body_bytes_sent "bodybytessent"http_referer” ’
# ‘“httpuseragent""http_user_agent" "httpuseragent""http_x_forwarded_for”’;
log_format json '{"@timestamp":"$time_iso8601",''"host":"$server_addr",''"clientip":"$remote_addr",''"remote_user":"$remote_user",''"request":"$request",''"http_user_agent":"$http_user_agent",''"size":$body_bytes_sent,''"responsetime":$request_time,''"upstreamtime":"$upstream_response_time",''"upstreamhost":"$upstream_addr",''"http_host":"$host",''"requesturi":"$request_uri",''"url":"$uri",''"domain":"$host",''"xff":"$http_x_forwarded_for",''"referer":"$http_referer",''"status":"$status"}';access_log logs/access.log json;sendfile on;keepalive_timeout 65;server {listen 80;server_name localhost;location / {root html;index index.html index.htm;}}
}
###########################
实际上就是添加了json格式的日志
log_format json ‘{"@timestamp":“KaTeX parse error: Double superscript at position 35: … '̲"host":"server_addr”,’
‘“clientip”:“KaTeX parse error: Double superscript at position 34: … '̲"remote_user":"remote_user”,’
‘“request”:“KaTeX parse error: Double superscript at position 30: … '̲"http_user_agen…http_user_agent”,’
‘“size”:KaTeX parse error: Double superscript at position 37: … '̲"responsetime":request_time,’
‘“upstreamtime”:“KaTeX parse error: Double superscript at position 45: … '̲"upstreamhost":…upstream_addr”,’
‘“http_host”:“KaTeX parse error: Double superscript at position 27: … '̲"requesturi":"request_uri”,’
‘“url”:“KaTeX parse error: Double superscript at position 26: … '̲"domain":"host”,’
‘“xff”:“KaTeX parse error: Double subscript at position 7: http_x_̲forwarded_for",…http_referer”,’
‘“status”:"$status"}’;
access_log logs/access.log json;
日志保存在/usr/local/nginx/logs/下
配置完成后，访问nginx。查看日志是否修改为json
[root@i4tnginx]# tail -f logs/access.log
{"@timestamp"“host”:“10.4.82.203”,“clientip”:“10.2.52.15”,“remote_user”:"-",“request”:“GET / HTTP/1.1”,“http_user_agent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/0101 Firefox/60.0”,“size”:0,“responsetime”:0.000,“upstreamtime”:"-",“upstreamhost”:"-",“http_host”:“10.4.82.203”,“requesturi”:"/",“url”:"/index.html",“domain”:“10.4.82.203”,“xff”:"-",“referer”:"-",“status”:“304”}
{"@timestamp"“host”:“10.4.82.203”,“clientip”:“10.2.52.15”,“remote_user”:"-",“request”:“GET / HTTP/1.1”,“http_user_agent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/0101 Firefox/60.0”,“size”:0,“responsetime”:0.000,“upstreamtime”:"-",“upstreamhost”:"-",“http_host”:“10.4.82.203”,“requesturi”:"/",“url”:"/index.html",“domain”:“10.4.82.203”,“xff”:"-",“referer”:"-",“status”:“304”}
测试logstash配置文件是否正常
我们尽量都是用efk用户，所以需要提前将logstash设置为efk的属主属组
chown -R efk.efk /usr/local/logstash
/usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/nginx.conf -t
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash’s logs to /usr/local/logstash/logs which is now configured via log4j2.properties
Configuration OK
[-01-28T11:54:38,481][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
在Nginx服务器上启动logstash
[root@abcdocker logstash]# su - efk
[efk@abcdocker ~]$ /usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/nginx.conf
温馨提示，一定要确保logs目录efk有权限写入，建议我们在启动efk之前在执行一次chown -R efk.efk /usr/local/logstash
请确保logstash中的file文件有读取权限，否则无法在ES中创建索引！
我们可以查一下索引
[root@YZSJHL82-203 local]# curl -XGET ‘200.200.100.51:9200/_cat/indices?v&pretty’
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 9l1XmifhTd2187a9Zpkqsw 1 1 1 0 3.2kb 3.2kb
yellow open pro_nginx_access-.02.15 Guze8x5hTymSzqzQKu5PTQ 5 1 1315 0 1.3mb 1.3mb
Kibana 配置
目前logstash已经将收集的日志存储在es里面，我们需要用kibana进行展示
查看索引命令
[root@YZSJHL82-203 local]# curl -XGET ‘200.200.100.51:9200/_cat/indices?v&pretty’
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 9l1XmifhTd2187a9Zpkqsw 1 1 1 0 3.2kb 3.2kb
yellow open pro_nginx_access-.02.15 Guze8x5hTymSzqzQKu5PTQ 5 1 1315 0 1.3mb 1.3mb
Kibana创建索引
创建完毕后查看索引
