文档章节

Using MQTTWarn to monitor server or device

openthings
 openthings
发布于 2015/05/25 08:59
字数 1207
阅读 157
收藏 1

How do your servers talk to you?

Most of us monitor what our servers do, in some way or another. We might use Icinga/Nagios  for some tasks, or use any of what feels like a myriad different tools  for other tasks. I assume, that in most cases, people use e-mail, SMS,  etc. for sending out warning or critical alerts when they occur.

Each of these monitoring services needs to be configured to send  alerts via a particular channel (e.g. SMTP, SMS) to particular users,  and if the need to change these associations arises, most people have to  reconfigure their software in order to do so. For Icinga/Nagios, there are "recipes" on getting particular notification systems configured; that works, of course, but it's painful, error-prone, and the configuration is very static.

Some specialized tools have other mechanisms. Ansible for example, has a slew of notification modules  (some of which I even wrote), which can announce to your deployment  team what's going on while it runs. Here again, each time you use a  particular notification module in Ansible, you have to configure it to  notify a particular service's account. What if that changes? Do you  really want to change all the Ansible playbooks ...? No.

Let me show you what I mean by listing the source of an alert, its type, and its destination:

icinga       disks          jane@example.com icinga       mail queue     jane@example.com icinga       dnsbl          SMS:1234567890 ansible      playbook       twilio:john18 ...

So, when Icinga reports a problem with a DNS blacklist, an SMS will  be sent, and when Icina warns about a mail queue, an e-mail to Jane is  fired off.

Suppose we want to change that: in addition to having an e-mail sent  off, we wish to alert a bunch of mobile phones via Pushover. Or we want  to change the content of the message, etc. For most of us that means  chasing down all instances of the configured notification method, and  modifying / adding to that. No thanks. Really not.

Can we set up some sort of notification "bus" which services use to  send notifications to, and which then decides, using a central config,  what to do with those messages, maybe even reformat them on a  per-destination basis? Yes we can.

mqttwarn

We created that recently. It's called mqttwarn, and it's pretty easy to set up. All you need is an MQTT broker, which is a lightweight "bus" for your network. Your services then "talk" to that broker (i.e. send it messages). mqttwarn  grabs those messages, and dispatches them on to a notification service  (SMTP, NNTP, Twilio, Pushover, HTTP, ...), stores them in files or in  MySQL, publishes them to Redis, pipes them to programs, etc. All  messages, some only (filtered by content, say).

I'll show you a small example to hopefully whet your appetite.

mqttwarn is configured in an ini-type file. The difficult part is setting up the so-called service targets which are notification services. I'll configure mqttwarn to provide a target called smtp:jplocal which delivers via SMTP, and another one called pushover:icinga which delivers a message to my mobile phone via Pushover.net.

[defaults] hostname  = 'localhost' port      = 1883 ; name the service providers you will be using. launch   = smtp, pushover [config:pushover] targets = {     'icinga'      : ['xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'],   } [config:smtp] server  =  'localhost:25' sender  =  "MQTTwarn <jpm@localhost>" starttls  =  False targets = {     'jplocal'     : [ 'jpm@localhost' ],     } [monitoring/+] targets = smtp:jplocal, pushover:icinga

When I launch mqttwarn, it will subscribe to the MQTT topic monitoring/+. (The + is a single-level wildcard which matches any one subtree.)

If I then publish a message to my MQTT broker, mqttwarn  will receive that message and act upon it. In this example I'll use a  command-line utility to do so, but we'd normally configure our  monitoring system (or Ansible, etc.) to publish via MQTT, and there are  all manner of language bindings to accomplish that.

mosquitto_pub -t monitoring/warning -m 'Disk utilization: 94%'

As you've probably guessed, -t specifies the topic name, and -m the payload to publish.

Do we have e-mail?

Date: Thu, 03 Apr 2014 09:26:36 +0200 From: MQTTwarn <jpm@localhost> To: jpm@localhost Subject: mqttwarn X-Mailer: mqttwarn Disk utilization: 94%

Simultaneously, mqttwarn notified the message via Pushover, and my phone alerts me accordingly:

mqttwarn on pushover

Coming back to Ansible, if I wanted to post that same notification from a playbook, I would use the mqtt notification module. (The strange-looking quotes are necessary to protect the colon in the string.)

--- - hosts: all   tasks:   ...   - local_action: 'mqtt                   topic=monitoring/warning                   payload="Disk utilization: 94%"'

If I later change my mind and want to modify one of the targets, say,  I want to add another e-mail recipient, store messages, etc., I just  modify mqttwarn's  configuration file; no need to change the system which actually  initiated the notification alert (i.e. I don't have to alter my Icinga configuration).

Looking at Icinga (or Nagios), we can very easily add something similar. Consider your fugly notification module

define command{     command_name    notify-service-by-email     command_line    /usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$     }

and now look at mine:

define command{     command_name notify-service-by-mqtt     command_line /usr/bin/notify-by-mqtt.py     }

The rather simple notify-by-mqtt program extracts the values I'm interested in from the environment (note: requires setting enable_environment_macros=1 in icinga.cfg) and produces a JSON payload which is published via MQTT to mqttwarn. The payload looks like this:

{     "_type": "icinga",      "date": "2014-04-04",      "eventstarttime": "1396621975",      "hostdisplayname": "localhost",      "hostname": "localhost",      "hoststatetype": "HARD",      "servicedesc": "JPtest",      "servicedisplayname": "JPtest",      "serviceoutput": "file /tmp/f1: ENOENT",      "servicestate": "CRITICAL",      "servicestateid": "2",      "shortdatetime": "2014-04-04 16:38:25",      "time": "16:38:25",      "timet": "1396622305" }

On the receiving end, mqttwarn  can filter messages, and I change their formatting on a per/service  basis (e.g. I can be more verbose in an e-mail than in a tweet). mqttwarn can transform data on the fly, and it can apply templates for format the message.

Let me show you a simple example. I'll take the above mqttwarn configuration and add the title and format lines to it:

[monitoring/+] targets = smtp:jplocal, pushover:icinga title = {servicestate} {servicedisplayname} format = {time}: {hostname}:{servicedisplayname} is {servicestate}: {serviceoutput}

mqttwarn attempts to  decode JSON from the payload received over MQTT, and it can use the  JSON elements to populate format-type strings. The result as seen on my  phone via Pushover is like this:

Seen from bottom to top:

  1. our first example

  2. an Icinga alert with the raw JSON in it

  3. the final result with formatted title and body

We've had a lot of very positive feedback on mqttwarn, and I'm confident you'll be able to put it to good use!


本文转载自:http://jpmens.net/2014/04/03/how-do-your-servers-talk-to-you/

openthings
粉丝 325
博文 1140
码字总数 689435
作品 1
东城
架构师
私信 提问
Kickstart Options

The following options can be placed in a kickstart file. If you prefer to use a graphical interfacefor creating your kickstart file, use the Kickstart Configurator application. ......

davis_qi
2014/08/21
123
0
18 Command Line Tools to Monitor Linux Performance

It’s really very tough job for every System or Network administrator to monitor and debug Linux System Performance problems every day. After being a Linux Administrator for 5 y......

wuyaoito
2014/02/18
0
0
在Android Studio下使用Hierarchy Viewer

前言 最近看到这篇文章《Android UI性能优化详解》,里面使用了Hierarchy Viewer来对布局进行优化。开发android这么久了,一直都有听过这个工具,但是重来都没真正去使用过,不得不说是一种失...

天王盖地虎626
03/18
133
0
Ubuntu 12.04 Eclipse 调试的时候到断点停住 就界面卡死

uname -a Linux XXXXXX 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ctrl+alt+f1,f2 也动不了 运行到断点 整个界面也就卡死了 以前不怎么......

sunyh
2013/10/23
1K
0
Centos 7 安装zabbix3.0

我是初学者,不会编译安装只能使用yum安装以下教程 以下有说错的地方请给位看完纠正一下,谢谢各位。 Install Zabbix 3.0 (Monitoring Server) on CentOS 7.x Zabbix is a free and open sou...

lg070224
2018/07/04
0
0

没有更多内容

加载失败,请刷新页面

加载更多

java通过ServerSocket与Socket实现通信

首先说一下ServerSocket与Socket. 1.ServerSocket ServerSocket是用来监听客户端Socket连接的类,如果没有连接会一直处于等待状态. ServetSocket有三个构造方法: (1) ServerSocket(int port);...

Blueeeeeee
今天
6
0
用 Sphinx 搭建博客时,如何自定义插件?

之前有不少同学看过我的个人博客(http://python-online.cn),也根据我写的教程完成了自己个人站点的搭建。 点此:使用 Python 30分钟 教你快速搭建一个博客 为防有的同学不清楚 Sphinx ,这...

王炳明
昨天
5
0
黑客之道-40本书籍助你快速入门黑客技术免费下载

场景 黑客是一个中文词语,皆源自英文hacker,随着灰鸽子的出现,灰鸽子成为了很多假借黑客名义控制他人电脑的黑客技术,于是出现了“骇客”与"黑客"分家。2012年电影频道节目中心出品的电影...

badaoliumang
昨天
14
0
很遗憾,没有一篇文章能讲清楚线程的生命周期!

(手机横屏看源码更方便) 注:java源码分析部分如无特殊说明均基于 java8 版本。 简介 大家都知道线程是有生命周期,但是彤哥可以认真负责地告诉你网上几乎没有一篇文章讲得是完全正确的。 ...

彤哥读源码
昨天
15
0
jquery--DOM操作基础

本文转载于:专业的前端网站➭jquery--DOM操作基础 元素的访问 元素属性操作 获取:attr(name);$("#my").attr("src"); 设置:attr(name,value);$("#myImg").attr("src","images/1.jpg"); ......

前端老手
昨天
7
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部