使用ELK在DC / OS中进行日志管理

原创
2017/04/25 14:43
阅读数 552

##参考 https://docs.mesosphere.com/1.9/monitoring/logging/aggregating/elk/

https://github.com/christtrc/dcos/blob/master/dcos-admin-logging/dcos-admin-logging-aggregationmd.md

略有改动

elk版本

5.2.1

安装

1.安装filebeat

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.2.1-x86_64.rpm

sudo rpm -vi filebeat-5.2.1-x86_64.rpm

2.创建/var/log/dcos目录

sudo mkdir -p /var/log/dcos

3.归集Master和Agent上的日志

Master

创建一个systemd服务,该服务解析DC/OS集群通过journalctl输出的日志并将其导向/var/log/dcos/dcos/dcos.log.

sudo tee /etc/systemd/system/dcos-journalctl-filebeat.service<<-EOF 
[Unit]
Description=DCOS journalctl parser to filebeat
Wants=filebeat.service
After=filebeat.service

[Service]
Restart=always
RestartSec=5
ExecStart=/bin/sh -c '/usr/bin/journalctl --no-tail -f \
  -u dcos-3dt.service \
  -u dcos-3dt.socket \
  -u dcos-adminrouter-reload.service \
  -u dcos-adminrouter-reload.timer   \
  -u dcos-adminrouter.service        \
  -u dcos-bouncer.service            \
  -u dcos-ca.service                 \
  -u dcos-cfn-signal.service         \
  -u dcos-cosmos.service             \
  -u dcos-download.service           \
  -u dcos-epmd.service               \
  -u dcos-exhibitor.service          \
  -u dcos-gen-resolvconf.service     \
  -u dcos-gen-resolvconf.timer       \
  -u dcos-history.service            \
  -u dcos-link-env.service           \
  -u dcos-logrotate-master.timer     \
  -u dcos-marathon.service           \
  -u dcos-mesos-dns.service          \
  -u dcos-mesos-master.service       \
  -u dcos-metronome.service          \
  -u dcos-minuteman.service          \
  -u dcos-navstar.service            \
  -u dcos-networking_api.service     \
  -u dcos-secrets.service            \
  -u dcos-setup.service              \
  -u dcos-signal.service             \
  -u dcos-signal.timer               \
  -u dcos-spartan-watchdog.service   \
  -u dcos-spartan-watchdog.timer     \
  -u dcos-spartan.service            \
  -u dcos-vault.service              \
  -u dcos-logrotate-master.service  \
  > /var/log/dcos/dcos.log 2>&1'
ExecStartPre=/usr/bin/journalctl --vacuum-size=10M

[Install]
WantedBy=multi-user.target
EOF

创建一个新的filebeat.yml配置文件,添加一条指向/var/log/dcos/dcos.log文件的新记录,该文件(后续步骤使用)用于归集DC/OS集群中的日志:

注: $LOGSTASH_HOSTNAME需要替换为实际地址.

sudo tee /etc/filebeat/filebeat.yml <<-EOF 
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/mesos/*.log
    - /var/log/dcos/dcos.log
  fields:
      node: Master
  fields_under_root: true
tail_files: true
output.logstash:
     hosts: ["$LOGSTASH_HOSTNAME"]
EOF

Agent

创建一个systemd服务,该服务解析DC/OS集群通过journalctl输出的日志并将其导向/var/log/dcos/dcos/dcos.log.
sudo tee /etc/systemd/system/dcos-journalctl-filebeat.service<<-EOF 
[Unit]
Description=DCOS journalctl parser to filebeat
Wants=filebeat.service
After=filebeat.service

[Service]
Restart=always
RestartSec=5
ExecStart=/bin/sh -c '/usr/bin/journalctl --no-tail -f      \
  -u dcos-3dt.service                      \
  -u dcos-logrotate-agent.timer            \
  -u dcos-3dt.socket                       \
  -u dcos-mesos-slave.service              \
  -u dcos-adminrouter-agent.service        \
  -u dcos-minuteman.service                \
  -u dcos-adminrouter-reload.service       \
  -u dcos-navstar.service                  \
  -u dcos-adminrouter-reload.timer         \
  -u dcos-rexray.service                   \
  -u dcos-cfn-signal.service               \
  -u dcos-setup.service                    \
  -u dcos-download.service                 \
  -u dcos-signal.timer                     \
  -u dcos-epmd.service                     \
  -u dcos-spartan-watchdog.service         \
  -u dcos-gen-resolvconf.service           \
  -u dcos-spartan-watchdog.timer           \
  -u dcos-gen-resolvconf.timer             \
  -u dcos-spartan.service                  \
  -u dcos-link-env.service                 \
  -u dcos-vol-discovery-priv-agent.service \
  -u dcos-logrotate-agent.service          \
  > /var/log/dcos/dcos.log 2>&1'
ExecStartPre=/usr/bin/journalctl --vacuum-size=10M

[Install]
WantedBy=multi-user.target
EOF

创建一个新的filebeat.yml配置文件,添加一条指向/var/log/dcos/dcos.log文件的新记录,该文件(后续步骤使用)用于归集DC/OS集群中的日志:

注: $LOGSTASH_HOSTNAME需要替换为实际地址.

sudo tee /etc/filebeat/filebeat.yml <<-EOF 
filebeat.prospectors:
- input_type: log
  paths:
    - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stdout
    - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stderr
    - /var/log/mesos/*.log
    - /var/log/dcos/dcos.log
  exclude_files:
              [
                "/var/lib/mesos/slave/slaves/*/frameworks/*/executors/[\\s\\S]*logstash[\\s\\S]*"
              ]
  fields:
      node: Agent
  fields_under_root: true
tail_files: true
output.logstash:
  # The Logstash hosts
     hosts: ["$LOGSTASH_HOSTNAME"]
EOF

注意:exclude_files中应当排除一些不要收集日志的应用地址.由于我在logstash中配置有stdout输出,为了防止出现死循环,所以我移除了logstash的stdout抓取

4.启用日志归集和Filebeat服务

sudo chmod 0755 /etc/systemd/system/dcos-journalctl-filebeat.service
sudo systemctl daemon-reload
sudo systemctl start dcos-journalctl-filebeat.service
sudo chkconfig dcos-journalctl-filebeat.service on
sudo systemctl start filebeat
sudo chkconfig filebeat on

5.logstash配置

注意:$ELASTICSEARCH_URL为实际elasticsearch地址.user和password为x-pack的账号和密码.

input{
     beats  {
              port => 5044
      }
}
filter{
    grok {
        patterns_dir => ["/etc/logstash/logstash-patterns"]
        match => { "source" => "%{TASKPATH}" }
    }
}
output{
    stdout{
        codec=>rubydebug
    }
  elasticsearch {
    user => 'x-pack账号'
    password => 'x-pack密码'
    hosts => ["$ELASTICSEARCH_URL]
    index => "dcos-other-%{+YYYY.MM.dd}"
    flush_size => 20000
    idle_flush_time => 10
    template_overwrite => true

  }
}

同时在/etc/logstash/logstash-patterns中添加

PATHELEM [^/]+
TASKPATH ^/var/lib/mesos/slave/slaves/%{PATHELEM:agent}/frameworks/%{PATHELEM:framework}/executors/%{PATHELEM:executor}/runs/%{PATHELEM:run}

在kibana中查看

图1

filebeat-kafka-logstash

修改filebeat配置,移除其他output(例如output.logstash),添加

output.kafka:
  hosts: ["broker.kafka.l4lb.thisdcos.directory:9092"]
  topic: "topic_elastic_collect"
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

官网:https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html

logstash修改

修改logstash.conf,添加kafka的input

 kafka {
    bootstrap_servers => "192.168.3.37:9383,192.168.1.71:9164,192.168.1.73:9312"
    group_id => "logstash"
    topics => ["topic_elastic_collect"]
    max_poll_records => "1000"
    codec => "json"
  }

官网: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html

##注意

  • 要在dc/os中的宿主机配置相对应的hostname,防止出现上图的beat.hostname都是 localhost.localdomain.导致没法区分日志来至哪一台宿主机
  • filebeat和logstash有stdout输出的时候,记得要在filebeat中排除,防止造成递归死循环
展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部