Beats数据采集

2018/04/30 13:33
阅读数 564

Beats数据采集

Beats是elastic公司的一款轻量级数据采集产品,它包含了几个子产品:

packetbeat(用于监控网络流量)、

filebeat(用于监听日志数据,可以替代logstash-input-file)、

topbeat(用于搜集进程的信息、负载、内存、磁盘等数据)、

winlogbeat(用于搜集windows事件日志)

另外社区还提供了dockerbeat等工具。由于他们都是基于libbeat写出来的,因此配置上基本相同,只是input输入的地方各有差异。

本文按照如下的内容依次进行介绍:

背景知识:关于Powershell的使用 packetbeat的下载、部署、使用、结果样例 filebeat的下载、部署、使用、样例 topbeat的样例 winlogbeat的样例

关于Powershell 如果你是想在linux下使用,那么可以跳过本节。

elastic中的Beats在windows环境中基本都是使用Powershell的脚本,因此用户必须对Powershell有一定的了解。Powershell可以理解成windows对命令行的高级封装,加了个壳,从而支持更多高级的用法。在windows7开始,系统就内置了Powershell工具。因此如果你的系统是xp这种比较老的版本,就需要自己安装Powershell了。

启动Powershell 在windows下,有两种方式打开Powershell(要以管理员的身份打开)。

通过图标打开 在windows下开启搜索,输入powershell,右键以管理员身份运行。

通过命令行启动 在系统路径C:\Windows\System32下,以管理员身份启动cmd.exe(右键选择 以管理员身份运行)。

输入命令Powershell,进入Powershell命令窗口。

C:\Windows\system32>Powershell Windows PowerShell 版权所有 (C) 2009 Microsoft Corporation。保留所有权利。

PS C:\Windows\system32> 开启脚本限制 默认的情况下,系统会禁止运行脚本,返回下面的错误提示:

PS E:\packetbeat> .\install-service-packetbeat.ps1 无法加载文件 E:\packetbeat\install-service-packetbeat.ps1,因为在此系统中禁止执 行脚本。有关详细信息,请参阅 "get-help about_signing"。 所在位置 行:1 字符: 33

  • .\install-service-packetbeat.ps1 <<<<
    • CategoryInfo : NotSpecified: (:) [], PSSecurityException
    • FullyQualifiedErrorId : RuntimeException 需要修改该参数执行下面的命令,开启Powershell脚本功能:

PS E:\packetbeat> set-ExecutionPolicy RemoteSigned Packetbeat 网络流量监控 Packetbeat属于beats产品的一部分,专门负责网络数据包分析,可以:

针对特定的网卡监听流量; 可以设置相关的监听对象和端口号,支持dns,http,memcahce,mysql,pgsql,redis,thrift,mongodb等; 可以输出到特定的目的地,如elasticsearch、logstash、file、console等。 下载 https://www.elastic.co/downloads/beats

部署 linux环境 第一步,解压缩 下载.tar.gz的安装包后,解压:

tar -zxvf packetbeat-1.2.3-x86_64.tar.gz 进入解压后的文件夹,里面有3个文件:

--- packetbeat #启动文件 --- packetbeat.template.json #Elasticsearch中的映射配置 --- packetbeat.yml #Packetbeat的配置文件 第二步,修改配置文件 配置文件包括了几大部分:

配置网络监听的显卡

interfaces: device:any

配置协议

protocols: http: ports:[80,8080,9000] redis: ports:[6379]

配置输出

output: elasticsearch: hosts:["localhost:9200"] inex:"packetbeat" template: name:"packetbeat" path:"packetbeat.template.json" overwrite:false logstah: hosts:["localhost:5044"] file: path:"/tmp/packetbeat" filename:packetbeat console: shipper: logging: 第三步,运行 正常的运行:

./packetbeat 如果想要后台运行,则可以像下面这样:

nohup ./packetbeat & 默认日志都会输出到nohup.out中。

windows环境 第一步,解压 相比linux,多了两个powershell的脚本。

--- install-service-packetbeat.ps1 # 注册脚本 --- uninstall-service-packetbeat.ps1 # 注销脚本 --- packetbeat.exe #启动文件 --- packetbeat.template.json #Elasticsearch中的映射配置 --- packetbeat.yml #Packetbeat的配置文件 第二步,以管理员身份进入命令行,运行注册脚本 进入指定的目录,运行注册脚本。

.\install-service-winlogbeat.ps1 第三步,启动服务 Start-Service packetbeat.exe 对接Elasticsearch Packetbeat配置如下:

elasticsearch: hosts: ["localhost:9200"] index: "packetbeat" template: name: "packetbeat" path: "packetbeat.template.json" 对接logstash Packetbeat配置如下:

logstash: # The Logstash hosts hosts: ["localhost:5044"] logstash采用logstash-input-beats接收,配置可以参考如下:

input{ beats{ port => 5044 } stdin{} } output{ stdout{ codec => rubydebug } file{ path => "E:\server.log" } } 存储到file PacketBeat配置:

file: path: "E:/packetbeat" filename: packetbeat 默认是按照文件大小轮询。

日志管理 日志可以设置输出的位置,以及级别。跟平常使用的log4j差不多:

logging: files: path: E:/mybeat name: mybeat level: debug Packetbeat监听到的内容 { "_index": "packetbeat-2016.08.01", "_type": "dns", "_id": "AVZELeQzbZnlZq0jh6Vk", "_version": 1, "_score": 1, "_source": { "@timestamp": "2016-08-01T03:37:53.106Z", "beat": { "hostname": "XINGHL", "name": "XINGHL" }, "bytes_in": 31, "bytes_out": 260, "client_ip": "10.4.45.44", "client_port": 51599, "client_proc": "", "client_server": "", "count": 1, "direction": "out", "dns": { "additionals": [ { "class": "IN", "data": "115.239.210.176", "name": "ns4.a.shifen.com", "ttl": 281, "type": "A" }, { "class": "IN", "data": "119.75.222.17", "name": "ns5.a.shifen.com", "ttl": 281, "type": "A" }, { "class": "IN", "data": "61.135.165.224", "name": "ns1.a.shifen.com", "ttl": 281, "type": "A" }, { "class": "IN", "data": "180.149.133.241", "name": "ns2.a.shifen.com", "ttl": 281, "type": "A" }, { "class": "IN", "data": "61.135.162.215", "name": "ns3.a.shifen.com", "ttl": 281, "type": "A" } ], "additionals_count": 5, "answers": [ { "class": "IN", "data": "www.a.shifen.com", "name": "sp1.baidu.com", "ttl": 33, "type": "CNAME" }, { "class": "IN", "data": "61.135.169.125", "name": "www.a.shifen.com", "ttl": 282, "type": "A" }, { "class": "IN", "data": "61.135.169.121", "name": "www.a.shifen.com", "ttl": 282, "type": "A" } ], "answers_count": 3, "authorities": [ { "class": "IN", "data": "ns5.a.shifen.com", "name": "a.shifen.com", "ttl": 1182, "type": "NS" }, { "class": "IN", "data": "ns1.a.shifen.com", "name": "a.shifen.com", "ttl": 1182, "type": "NS" }, { "class": "IN", "data": "ns3.a.shifen.com", "name": "a.shifen.com", "ttl": 1182, "type": "NS" }, { "class": "IN", "data": "ns2.a.shifen.com", "name": "a.shifen.com", "ttl": 1182, "type": "NS" }, { "class": "IN", "data": "ns4.a.shifen.com", "name": "a.shifen.com", "ttl": 1182, "type": "NS" } ], "authorities_count": 5, "flags": { "authoritative": false, "recursion_allowed": true, "recursion_desired": true, "truncated_response": false }, "id": 32509, "op_code": "QUERY", "question": { "class": "IN", "name": "sp1.baidu.com", "type": "A" }, "response_code": "NOERROR" }, "ip": "210.83.210.155", "method": "QUERY", "port": 53, "proc": "", "query": "class IN, type A, sp1.baidu.com", "resource": "sp1.baidu.com", "responsetime": 1, "server": "", "status": "OK", "transport": "udp", "type": "dns" } } filebeat 日志监听 filebeat是Beats的重要组成部分,它可以作为轻量级的数据采集引擎,替代之前的logstash-forward。

下载 https://www.elastic.co/downloads/beats

说明 filebeat.yml为filebeat的配置文件,包括下面几个部分:

-- filebeat # 配置filebeat监听的对象,即文件路径或者目录的路径 -- output # 输出配置,支持es,logstash,file,console等 -- shipper -- logging # 配置日志 filebeat.template.json 为默认提供的elasticsearch映射模板 filebeat为主要的执行程序

运行 linux环境 运行命令解压安装包——filebeat.tar.gz

tar -zxvf filebeat.tar.gz 编辑filebeat.yml

vim filebeat.yml 启动filebeat

nohup ./filebeat & windows环境 以管理员身份运行cmd, 并执行Powershell命令,进入PS模式.启动filebeat注册脚本:

C:\Windows\system32>Powershell Windows PowerShell 版权所有 (C) 2009 Microsoft Corporation。保留所有权利。

PS C:\Windows\system32> e: PS E:> cd .\filebeat-1.2.3-windows PS E:\filebeat-1.2.3-windows> dir

目录: E:\filebeat-1.2.3-windows

Mode LastWriteTime Length Name


----- 2016/5/18 4:33 10361856 filebeat.exe ----- 2016/5/18 4:33 814 filebeat.template.json ----- 2016/5/18 4:33 17533 filebeat.yml ----- 2016/5/18 4:33 442 install-service-filebeat.ps1 ----- 2016/5/18 4:33 184 uninstall-service-filebeat.ps1

PS E:\filebeat-1.2.3-windows> .\install-service-filebeat.ps1

Status Name DisplayName


Stopped filebeat filebeat

编辑配置文件,filebeat.yml

启动filebeat文件

PS E:\filebeat-1.2.3-windows> Start-Service filebeat 样例 { "_index": "filebeat-2016.08.01", "_type": "log", "_id": "AVZE1AMfbZnlZq0jh6cF", "_version": 1, "_score": 1, "_source": { "@timestamp": "2016-08-01T06:39:15.193Z", "beat": { "hostname": "XINGHL", "name": "XINGHL" }, "count": 1, "fields": null, "input_type": "log", "message": "hello filebeat", "offset": 22988, "source": "e:\logs\test.log", "type": "log" } } topbeat 监听进程资源信息 启动方式与前面几种类似,这里就不过多赘述了。

topbeat - windows版 { "_index": "topbeat-windows-2016.08.01", "_type": "process", "_id": "AVZE7zC6bZnlZq0jh8QD", "_version": 1, "_score": 1, "_source": { "@timestamp": "2016-08-01T07:09:01.206Z", "beat": { "hostname": "XINGHL", "name": "XINGHL" }, "count": 1, "proc": { "cmdline": "%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16", "cpu": { "user": 5538, "user_p": 0, "system": 7753, "total": 13291, "start_time": "Jan01" }, "mem": { "size": 3391488, "rss": 3366912, "rss_p": 0, "share": 0 }, "name": "csrss.exe", "pid": 544, "ppid": 0, "state": "running", "username": "NT AUTHORITY\SYSTEM" }, "type": "process" } } topbeat - linux版本 { "_index": "topbeat-2016.08.01", "_type": "process", "_id": "AVZE6Mh4bZnlZq0jh6jT", "_version": 1, "_score": 1, "_source": { "@timestamp": "2016-08-01T07:01:09.641Z", "beat": { "hostname": "10.0.67.101", "name": "10.0.67.101" }, "count": 1, "proc": { "cpu": { "user": 0, "user_p": 0, "system": 0, "total": 0, "start_time": "Jul06" }, "mem": { "size": 0, "rss": 0, "rss_p": 0, "share": 0 }, "name": "migration/0", "pid": 5, "ppid": 2, "state": "sleeping", "username": "root" }, "type": "process" } } winlogbeat windows事件监听 启动方式与前面几种类似,这里就不过多赘述了。

{ "_index": "winlogbeat-2015.11.09", "_type": "wineventlog", "_id": "AVZE_J7FbZnlZq0jh_sL", "_version": 1, "_score": 1, "_source": { "@timestamp": "2015-11-09T00:28:50.953Z", "beat": { "hostname": "XINGHL", "name": "XINGHL" }, "computer_name": "xinghailong", "count": 1, "event_id": 35, "level": "信息", "log_name": "System", "message": "时间服务现在用时间源 time.neusoft.com,0x9 (ntp.m|0x9|0.0.0.0:123->202.118.6.8:123) 同步系统时间。", "record_number": "25479", "source_name": "Microsoft-Windows-Time-Service", "type": "wineventlog", "user": { "domain": "NT AUTHORITY", "identifier": "S-1-5-19", "name": "LOCAL SERVICE", "type": "Well Known Group" } } } 参考 1 官方文档

2 ELK Beats文档

博客不再维护,请关注新的博客:https://xinghalo.github.io/

http://www.cnblogs.com/xing901022/category/677227.html http://www.cnblogs.com/xing901022/category/642865.html

展开阅读全文
加载中

作者的其它热门文章

打赏
0
0 收藏
分享
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部
返回顶部
顶部