文档章节

ELK6.1.3 stack for rhel6.8

雁南飞丶
 雁南飞丶
发布于 2018/02/08 17:28
字数 1695
阅读 94
收藏 0

简单拓扑图

IP HOSTNAME 运行服务
9.1.6.140 ES1

Kibana6.1.3  分析展示

Elasticsearch6.1.3 集群运算

Logstash6.1.3收集apache日志输出至redis

9.1.8.123 ES2

Elasticsearch6.1.3集群运算

Redis3.2  消息发布订阅(接收logstash客户端的消息,推送至ES)

9.1.8.153 ES3

Elasticsearch6.1.3集群运算

Logstash6.1.3(input来自redis的日志,output为elasticsearch服务器)

一、下载redis简单安装用来存放logstash收集的数据

wget http://download.redis.io/releases/redis-3.2.11.tar.gz
[root@ AOS2 @AutoTest01:/root]#tar xvf redis-3.2.11.tar.gz -C /usr/local/

[root@ AOS2 @AutoTest01:/usr/local]#ln -sv redis-3.2.11 redis
"redis" -> "redis-3.2.11"
[root@ AOS2 @AutoTest01:/usr/local]#cd redis

[root@ AOS2 @AutoTest01:/usr/local/redis]#make
......

启动
[root@ AOS2 @AutoTest01:/usr/local/redis/src]#./redis-server &
[1] 5701
[root@ AOS2 @AutoTest01:/usr/local/redis/src]#5701:C 08 Feb 17:19:59.985 # Warning: no config file specified, using the default config. In order to specify a config file use ./redis-server /path/to/redis.conf
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.2.11 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 5701
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

5701:M 08 Feb 17:19:59.987 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
5701:M 08 Feb 17:19:59.987 # Server started, Redis version 3.2.11
5701:M 08 Feb 17:19:59.987 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
5701:M 08 Feb 17:19:59.987 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
5701:M 08 Feb 17:19:59.987 * DB loaded from disk: 0.000 seconds
5701:M 08 Feb 17:19:59.987 * The server is now ready to accept connections on port 6379

监听端口6379

[root@ AOS2 @AutoTest01:/root]#netstat -antlp |grep :6379
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      5701/./redis-server 
tcp        0      0 :::6379                     :::*                        LISTEN      5701/./redis-server 



[root@ AOS2 @AutoTest01:/usr/local/redis/src]#./redis-cli 
127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379> get foo
"bar"
127.0.0.1:6379> 



OK可以简单使用了

二、配置logstash 为agent端为output 参考logstash output插件https://www.elastic.co/guide/en/logstash/5.5/index.html

logstash代理收集数据输出至redis服务器

[root@zdhdbbsj conf.d]# cat apachelog.conf 
input {
    file {
	    path    => ["/var/log/httpd/access_log"]
	    type    => "apachelog"
	    start_position => "beginning"
	    }
	}

filter {
	  grok {
	   match => { "message" => "%{COMBINEDAPACHELOG}" }
	    }
	}
output {
	 redis {
		port	=> 6379
		host	=> ["9.1.8.123"]
		data_type	=> "list"
		db	=> 0
		key => "logstash-apachelog"
	}
}



[root@zdhdbbsj conf.d]# logstash -f ./apachelog.conf -t
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK

访问apache网页,生成访问日志,查看redis上的数据

[root@ AOS2 @AutoTest01:/usr/local/redis/src]#./redis-cli 

127.0.0.1:6379> LLEN logstash-apachelog
(integer) 20
127.0.0.1:6379> LINDEX logstash-apachelog 1
"{\"ident\":\"-\",\"@version\":\"1\",\"message\":\"9.5.7.222 - - [09/Feb/2018:10:13:21 +0800] \\\"GET / HTTP/1.1\\\" 403 3985 \\\"-\\\" \\\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\\\"\",\"type\":\"apachelog\",\"auth\":\"-\",\"timestamp\":\"09/Feb/2018:10:13:21 +0800\",\"request\":\"/\",\"host\":\"zdhdbbsj\",\"verb\":\"GET\",\"httpversion\":\"1.1\",\"referrer\":\"\\\"-\\\"\",\"agent\":\"\\\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\\\"\",\"bytes\":\"3985\",\"response\":\"403\",\"clientip\":\"9.5.7.222\",\"@timestamp\":\"2018-02-09T03:13:27.618Z\",\"path\":\"/var/log/httpd/access_log\"}"
127.0.0.1:6379> 
127.0.0.1:6379> LINDEX logstash-apachelog 0
"{\"ident\":\"-\",\"@version\":\"1\",\"message\":\"9.5.7.222 - - [09/Feb/2018:10:13:13 +0800] \\\"GET / HTTP/1.1\\\" 304 - \\\"-\\\" \\\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\\\"\",\"type\":\"apachelog\",\"auth\":\"-\",\"timestamp\":\"09/Feb/2018:10:13:13 +0800\",\"request\":\"/\",\"host\":\"zdhdbbsj\",\"verb\":\"GET\",\"httpversion\":\"1.1\",\"referrer\":\"\\\"-\\\"\",\"agent\":\"\\\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\\\"\",\"response\":\"304\",\"clientip\":\"9.5.7.222\",\"@timestamp\":\"2018-02-09T03:13:27.613Z\",\"path\":\"/var/log/httpd/access_log\"}"
127.0.0.1:6379> 

三、logstash配置为server端从redis读取数据发送elasticsearch或者集群

[root@zdhhamqb ~]# cat /etc/logstash/conf.d/server.conf 
input {
	 redis {
		port	=> 6379
		host	=> ["9.1.8.123"]
		data_type	=> "list"
		db	=> 0
		key => "logstash-apachelog"
	}
}
output {
    elasticsearch {
		index => "logstash-%{+YYYY.MM.dd}"
        port	=> 9200
		host	=> ["9.1.8.153"]
            }
	
}

刷新apache日志查看ES里面得索引日志

[root@zdhhamqb ~]# curl  -XGET 'http://9.1.8.153:9200/_cat/indices'
green open logstash-2018.02.09 d_u4lgEHRDyQALLjiWwbIw 5 1 10 0 161.7kb 86.7kb
green open students            fdfkpZVmTOeCRTxmSJXFAA 5 1  2 0  23.6kb 11.8kb
[root@zdhhamqb ~]# curl  -XGET 'http://9.1.8.153:9200/logstash-2018.02.09/_search?pretty'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 10,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logstash-2018.02.09",
        "_type" : "doc",
        "_id" : "LJXCeGEBXsK3Z7fytDfA",
        "_score" : 1.0,
        "_source" : {
          "type" : "apachelog",
          "clientip" : "9.5.7.222",
          "httpversion" : "1.1",
          "@timestamp" : "2018-02-09T04:13:06.800Z",
          "message" : "9.5.7.222 - - [09/Feb/2018:12:13:06 +0800] \"GET / HTTP/1.1\" 403 3985 \"-\" \"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\"",
          "@version" : "1",
          "ident" : "-",
          "verb" : "GET",
          "request" : "/",
          "timestamp" : "09/Feb/2018:12:13:06 +0800",
          "auth" : "-",
          "agent" : "\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\"",
          "host" : "zdhdbbsj",
          "referrer" : "\"-\"",
          "bytes" : "3985",
          "response" : "403",
          "path" : "/var/log/httpd/access_log"
        }
      },
      {
        "_index" : "logstash-2018.02.09",
        "_type" : "doc",
        "_id" : "JpXCeGEBXsK3Z7fysDfU",
        "_score" : 1.0,
        "_source" : {
          "type" : "apachelog",
          "clientip" : "9.5.7.222",
          "httpversion" : "1.1",
          "@timestamp" : "2018-02-09T04:13:05.796Z",
          "message" : "9.5.7.222 - - [09/Feb/2018:12:13:05 +0800] \"GET / HTTP/1.1\" 403 3985 \"-\" \"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\"",
          "@version" : "1",
          "ident" : "-",
          "verb" : "GET",
          "request" : "/",
          "timestamp" : "09/Feb/2018:12:13:05 +0800",
          "auth" : "-",
          "agent" : "\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/8.0; .NET4.0C; .NET4.0E; InfoPath.3)\"",
          "host" : "zdhdbbsj",
          "referrer" : "\"-\"",
          "bytes" : "3985",
          "response" : "403",
          "path" : "/var/log/httpd/access_log"
        }
      },

安装kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.0-x86_64.rpm

rpm -ivh kibana-6.2.0-x86_64.rpm

配置

[root@zdhdbbsj ~]# grep -v '^#' /etc/kibana/kibana.yml  |grep -v '^$'
server.port: 5601
server.host: "9.1.6.140"
server.name: "zdhdbbsj"
elasticsearch.url: "http://9.1.6.140:9200"

启动

[root@zdhdbbsj ~]# /etc/init.d/kibana status
kibana is running
监听端口5601
[root@zdhdbbsj ~]# netstat -antlp |grep :5601
tcp        0      0 9.1.6.140:5601              0.0.0.0:*                   LISTEN      20044/node          
[root@zdhdbbsj ~]# 

这个新版本的kibana好像不匹配

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.1.3-x86_64.rpm

[root@zdhdbbsj ~]# rpm -ivh kibana-6.1.3-x86_64.rpm 
warning: kibana-6.1.3-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                ########################################### [100%]
   1:kibana                 ########################################### [100%]


[root@zdhdbbsj kibana]# tail -f /var/log/kibana/kibana.stdout 
{"type":"response","@timestamp":"2018-02-09T06:21:54Z","tags":[],"pid":20044,"method":"get","statusCode":304,"req":{"url":"/ui/favicons/favicon-32x32.png","method":"get","headers":{"host":"9.1.6.140:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://9.1.6.140:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"zh-CN,zh;q=0.9,en;q=0.8","if-none-match":"\"8e183c2e644fb050707d89402e1f7a120a95e4d2\"","if-modified-since":"Thu, 01 Feb 2018 17:43:13 GMT"},"remoteAddress":"9.5.7.222","userAgent":"9.5.7.222","referer":"http://9.1.6.140:5601/app/kibana"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /ui/favicons/favicon-32x32.png 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2018-02-09T06:21:54Z","tags":[],"pid":20044,"method":"get","statusCode":304,"req":{"url":"/ui/favicons/favicon-16x16.png","method":"get","headers":{"host":"9.1.6.140:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36","accept":"image/webp,image/apng,image/*,*/*;q=0.8","referer":"http://9.1.6.140:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"zh-CN,zh;q=0.9,en;q=0.8","if-none-match":"\"13b869be5df4bdc56920edc16a28e67a7c08203b\"","if-modified-since":"Thu, 01 Feb 2018 17:43:13 GMT"},"remoteAddress":"9.5.7.222","userAgent":"9.5.7.222","referer":"http://9.1.6.140:5601/app/kibana"},"res":{"statusCode":304,"responseTime":1,"contentLength":9},"message":"GET /ui/favicons/favicon-16x16.png 304 1ms - 9.0B"}
{"type":"log","@timestamp":"2018-02-09T06:32:26Z","tags":["plugins","warning"],"pid":1001,"path":"/usr/share/kibana/src/core_plugins/vega","message":"Skipping non-plugin directory at /usr/share/kibana/src/core_plugins/vega"}
{"type":"log","@timestamp":"2018-02-09T06:32:27Z","tags":["status","plugin:kibana@6.1.3","info"],"pid":1001,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-09T06:32:27Z","tags":["status","plugin:elasticsearch@6.1.3","info"],"pid":1001,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-09T06:32:27Z","tags":["status","plugin:console@6.1.3","info"],"pid":1001,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-09T06:32:27Z","tags":["status","plugin:elasticsearch@6.1.3","info"],"pid":1001,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-02-09T06:32:27Z","tags":["status","plugin:metrics@6.1.3","info"],"pid":1001,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-09T06:32:27Z","tags":["status","plugin:timelion@6.1.3","info"],"pid":1001,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-09T06:32:27Z","tags":["listening","info"],"pid":1001,"message":"Server running at http://9.1.6.140:5601"}

 

访问http://9.1.6.140:5601

© 著作权归作者所有

雁南飞丶
粉丝 37
博文 176
码字总数 228466
作品 0
西安
运维
私信 提问
rhel6配置多用户tiger vnc server

OS:RHEL6.8 1 安装 yum install tigervnc-server 2 配置 编辑/etc/sysconfig/vncserver文件,如下 more /etc/sysconfig/vncservers VNCSERVERS="1:root" ==1是vnc服务,root是用户名 VNCSER......

hashing123
2018/08/07
0
0
shell自动备份文件或目录到本地和远程主机

#!/bin/bash #Function:自动备份给定列表中的目录或文件,并且可以保留N天备份的档案。 #可备份至远程主机指定的目录下,但需本机能免密码登录到远程主机,用到ssh-keygen #该脚本分为两个文件:...

zzjiwang
2016/11/12
0
0
数据结构 栈 c++ 源代码实现

#include #include #define STACKINITSIZE 10 #define STACKINCREMENTSIZE 10 #define Status bool #define OK true #define ERROR false typedef int SElemType; typedef struct SqStack{ S......

忘记江南
2017/11/26
0
0
链栈的实现

include include include typedef int DataType; typedef struct node{struct node *next;//指针域DataType data;//数据域}Stack; void initStack(Stack **head);//初始化 int isEmpty(Stac......

南桥北木
2018/02/24
0
0
关于java 内存溢出

高并发下java 内存溢出 0x0000000dbe636000 JavaThread "Connection evictor" daemon [_thread_in_vm, id=288304, stack(0x0000001defc90000,0x0000001defd90000)] 0x0000001d14513000 Java......

chunf95
2018/09/18
78
0

没有更多内容

加载失败,请刷新页面

加载更多

GitOps 初探

前言 GitOps 的概念最初来源于 Weaveworks 的联合创始人 Alexis 在 2017 年 8 月发表的一篇博客 GitOps - Operations by Pull Request。文章介绍了 Weaveworks 的工程师如何以 Git 作为事实的...

阿里云云栖社区
17分钟前
0
0
keytool生成密钥

生成JKS Java KeyStore文件 我们首先使用命令行工具keytool生成密钥 - 更具体地说.jks文件: keytool -genkeypair -alias mytest -keyalg RSA ...

Canaan_
19分钟前
0
0
【从入门到放弃-Java】并发编程-NIO使用

前言 上文【从入门到放弃-SpringBoot】SpringBoot源码分析-请求过程中我们了解到,tomcat接收、返回请求的过程都是基于NIO实现的。日常工作中有很多基于NIO的使用,我们知道NIO可以提高系统的...

阿里云官方博客
20分钟前
2
0
mysql 如何删除数据库中所有的表

SELECT concat('DROP TABLE IF EXISTS ', table_name, ';') FROM information_schema.tables WHERE table_schema = 'mydb'; mydb换成你想删除的数据库的名字......

RobertZhou
21分钟前
1
0
pytorch各种版本下载

链接地址:https://pytorch.org/get-started/previous-versions/

云烟成雨forever
21分钟前
0
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部