CentOS ELK5.0日志分析平台搭建

原创
2017/05/22 17:40
阅读数 384

之前我们在Windows平台上安装过Elasticsearch+X-Pack+Kibana工具(具体参考:Windows 安装Elasticsearch&Kibana&X-Pack),这里我们在Linux系统中做一个日志分析平台。

一.安装Elasticsearch

#wget -c https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.tar.gz
#tar -zxvf elasticsearch-5.4.0.tar.gz
#mkdir  /usr/elk
#mv elasticsearch-5.4.0 /usr/elk/elasticsearch
#cd /usr/elk/elasticsearch/bin

然后我们可以使用如下命令启动

./elasticsearch

启动过程中可能遇到如下问题

①.jvm内存不足

解决方法,修改如下配置文件,调整-Xms2g -Xmx2g为-Xms1g -Xmx1g

vim /usr/elk/elasticsearch/config/jvm.options

②.要求openjdk版本至少1.8,oracle jdk 1.7

这个时候需要升级openjdk或者使用oracle jdk替换

#查询本地安装的数据库,并且将其删除

#rpm -qa | grep jdk  
java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.x86_64
java-1.7.0-openjdk-1.7.0.0-1.32.1.11.1.el6.x86_64 

#yum remove java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.x86_64 
#yum remove java-1.7.0-openjdk-1.7.0.0-1.32.1.11.1.el6.x86_64

#当然,如果你觉得上述操作麻烦,建议直接使用如下方式

#yum remove java-1.6.0-openjdk*
#yum remove java-1.7.0-openjdk*

#然后通过如下方式检索java-1.8.0-openjdk

#yum search java-1.8.0-openjdk
============== N/S Matched: java-1.8.0-openjdk ==============
java-1.8.0-openjdk.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-debug.x86_64 : OpenJDK Runtime Environment with full debug on
java-1.8.0-openjdk-demo.x86_64 : OpenJDK Demos
java-1.8.0-openjdk-demo-debug.x86_64 : OpenJDK Demos with full debug on
java-1.8.0-openjdk-devel.x86_64 : OpenJDK Development Environment
java-1.8.0-openjdk-devel-debug.x86_64 : OpenJDK Development Environment with full debug on
java-1.8.0-openjdk-headless.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-headless-debug.x86_64 : OpenJDK Runtime Environment with full debug on
java-1.8.0-openjdk-javadoc.noarch : OpenJDK API Documentation
java-1.8.0-openjdk-javadoc-debug.noarch : OpenJDK API Documentation for packages with debug on
java-1.8.0-openjdk-src.x86_64 : OpenJDK Source Bundle
java-1.8.0-openjdk-src-debug.x86_64 : OpenJDK Source Bundle for packages with debug on

一般都能检索出来,如果检索不出来,建议去下载安装oracle jdk

#如果检索到java-1.8.0-openjdk,直接安装即可

#yum install  java-1.8.0-openjdk java-1.8.0-openjdk-devel java-1.8.0-openjdk-headless

③.root用户不允许运行

修改运行文件

#groupadd elkstack                       #创建组
#useradd elkstack -g elkstack -d /usr/elk -s /bin/bash  #创建用户
#passwd  elkstack                        #给用户创建密码
#chown -R elkstack:elkstack /usr/elk     #目录的拥有者

#########################################################
#cd /etc/skel/                           #进入用户登录状态管理目录,如果不执行此操作,则登录界面在sh中
#ls -a  
. ..  .bash_logout  .bash_profile .bashrc  .mozilla  
#复制文件到新用户的创建目录
#cp .bash_logout  /home/mysql/  
#cp .bash_profile  /home/mysql/  
#cp .bashrc  /home/mysql
#cd /

#########################################################
#su -l elkstack                          #切换用户 
#./elasticsearch/bin/elasticsearch       #启动elasticsearch 

④.当前用户创建文件的大小,可使用的内存有限制

max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
max number of threads [1024] for user [elkstack] is too low, increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Q:max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

#vi /etc/security/limits.conf  
添加如下内容:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
Q:max number of threads [1024] for user [elkstack] is too low, increase to at least [2048]

#vi /etc/security/limits.d/90-nproc.conf 

修改如下内容:
* soft nproc 1024

#修改为
* soft nproc 2048
Q:max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

#vi /etc/sysctl.conf 

添加下面配置:
vm.max_map_count=655360

system call filters failed to install; check the logs and fix your configuration or 
disable system call filters at your own risk

#vim config/elasticsearch.yml

添加
bootstrap.system_call_filter: false 

配置完成之后,执行命令

sysctl -p

以上是安装过程中遇到的比较多的问题

 

使用curl 检查是否成功启动

#curl -i http://127.0.0.1:9200

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 327

{
  "name" : "rGlFyHB",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "7sEFicrvQW-RPbJTjekbHg",
  "version" : {
    "number" : "5.4.0",
    "build_hash" : "780f8c4",
    "build_date" : "2017-04-28T17:43:27.229Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.0"
  },
  "tagline" : "You Know, for Search"
}

二.安装kibana

#wget -c https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz
#tar -zxvf kibana-5.4.0-linux-x86_64.tar.gz
#mv kibana-5.4.0-linux-x86_64 /usr/elk/kibana
#cd /usr/elk/
#chown -R elkstack:elkstack kibana
#./kibana/bin/kibana

使用curl检测是否成功启动(注意:必须先启动elasticsearch)

#curl -i http://localhost:5601

HTTP/1.1 200 OK
kbn-name: kibana
kbn-version: 5.4.0
cache-control: no-cache
content-type: text/html; charset=utf-8
content-length: 217
accept-ranges: bytes
Date: Mon, 22 May 2017 06:45:26 GMT
Connection: keep-alive

<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}

此外,kibana中需要配置elasticsearch的信息,如果elasticsearch的访问信息更新了,同样也需要更新kibana中的配置信息

#vim kibana/config/kibana.yml
#elasticsearch默认配置信息如下
elasticsearch.url: "http://localhost:9200"

三.安装logstash

#wget -c https://artifacts.elastic.co/downloads/logstash/logstash-5.4.0.tar.gz
#tar -zxvf logstash-5.4.0.tar.gz
#mv logstash-5.4.0 /usr/elk/logstash
#cd /usr/elk/
#chown -R elkstack:elkstack logstash
#./logstash/bin/logstash

测试是否安装成功

注意:最好以root用户运行,或者在sudoers中添加用户的sudo命令权限,否则可能产生好多问题

#./logstash/bin/logstash -e 'input{stdin{}} output{stdout{}}'

#启动之后,执行如下操作,测试是否有回显,如果有回显,则表示正确
#Hello World

 

四.配置&插件安装

①.远程访问

kibana.yml:

server.port:  5601
server.host:  "192.168.1.210"
elasticsearch.url: "http://192.168.1.210:9200"

elasticsearch.yml:

network.host: "192.168.1.210"
http.port: 9200

#加入新集群时使用的ip地址,默认是回环地址
#discovery.zen.ping.unicast.hosts: ["192.168.1.210"]  
#集群中最少的master数量
#discovery.zen.minimum_master_nodes: 3  

#bootstrap.system_call_filter: false 

logstash.yml

 http.host: "172.20.11.62"

②.安装X-Pack

注意:安装前必须停止elasticsearch与kibana服务

#cd /usr/elk/elasticsearch/bin
#./elasticsearch-plugin install x-pack

#cd /usr/elk/kibana/bin
#./kibana-plugin install x-pack

安装X-Pack完成之后,穷elasticsearchkibana ,会进行用户登录校验,默认用户名和密码如下

username : elastic
passowrd : changeme

但是,对于logstash来说,需要在配置文件中配置用户名才行,否则无法链接到elasticsearch

input {
  file {
    type =>"syslog"
     path => ["/var/log/messages", "/var/log/secure" ]
  }
  syslog {
    type =>"syslog"
    port =>"5544"
  }
}
output {
  stdout { codec=> rubydebug }
  elasticsearch {
	hosts => ["192.168.1.210:9200"]
	 user => elastic
         password => changeme
         index => "syslogstash-%{+YYYY.MM.dd}"
         template_overwrite => true	
	}
}

③.Kibana创建索引

Kibana创建索引的前提是logstash的pipline配置文件中存索引,并且logstash已经向elasticsearch注册了索引

index => "syslogstash-%{+YYYY.MM.dd}"
input {
  file {
    type =>"syslog"
     path => ["/var/log/messages", "/var/log/secure" ]
  }
  syslog {
    type =>"syslog"
    port =>"5544"
  }
}
output {
  stdout { codec=> rubydebug }
  elasticsearch {
	hosts => ["192.168.1.210:9200"]
	 user => elastic
         password => changeme
         index => "logstash-%{+YYYY.MM.dd}"
         template_overwrite => true	
	}
}

检测配置文件是否正确

#logstash/bin/logstash -f test_logstash.conf -t

启动logstash

#logstash/bin/logstash -f test_logstash.conf 

触发Input Event,让logstash主动注册index到elasticsearch

#logger -p info "hello, remote rsyslog"

然后登录Kibana,点击Management->Index Patterns打开索引注册页面,点击左侧菜单栏中的【+】,新增索引。

如果你看不到Create按钮,那么很可能意味着索引没有注册成功,注册可能需要一个Input Event输入触发才行。

如果索引注册成功,那么点击Kibana菜单Discover,选择syslogstash-*索引,便能看到相应的事件。

参考:

ELK集群部署及收集nginx日志

Elasticsearch5.0 版本安装错误

记录Linux下安装elasticSearch时遇到的一些错误

centos7虚拟机安装elasticsearch5.0.x-安装篇

ElasticSearch 5.0.0 安装部署常见错误或问题

已解决:登录Linux的 -bash-4.2$ 问题

 

 

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部