文档章节

ELK简单安装测试

o
 osc_gu9d45li
发布于 2019/04/09 20:06
字数 7857
阅读 17
收藏 0

「深度学习福利」大神带你进阶工程师,立即查看>>>

<div class="output_wrapper" id="output_wrapper_id" style="font-size: 15px; color: rgb(62, 62, 62); line-height: 1.8; word-spacing: 2px; letter-spacing: 2px; font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif; background-image: linear-gradient(90deg, rgba(50, 0, 0, 0.05) 3%, rgba(0, 0, 0, 0) 3%), linear-gradient(360deg, rgba(50, 0, 0, 0.05) 3%, rgba(0, 0, 0, 0) 3%); background-size: 20px 20px; background-position: center center;"><h2 id="h1" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.4em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">1 介绍组件</span></h2> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">Filebeat是一个日志文件托运工具,在你的服务器上安装客户端后,filebeat会监控日志目录或者指定的日志文件,追踪读取这些文件(追踪文件的变化,不停的读)。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">Kafka是一种高吞吐量的分布式发布订阅消息系统,它可以处理消费者规模的网站中的所有动作流数据。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">ElasticSearch它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口。Elasticsearch 中的 Index 是一组具有相似特征的文档集合,类似于关系数据库模型中的数据库实例,Index 中可以指定 Type 区分不同的文档,类似于数据库实例中的关系表,Document 是存储的基本单位,都是 JSON 格式,类似于关系表中行级对象。我们处理后的 JSON 文档格式的日志都要在 Elasticsearch 中做索引,相应的 Logstash 有 Elasticsearch output 插件,对于用户是透明的。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">Hadoop 生态圈为大规模数据集的处理提供多种分析功能,但实时搜索一直是 Hadoop 的软肋。如今,Elasticsearch for Apache Hadoop(ES-Hadoop)弥补了这一缺陷,为用户整合了 Hadoop 的大数据分析能力以及 Elasticsearch 的实时搜索能力。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">Logstash 是一种功能强大的信息采集工具,类似于 Hadoop 生态圈里的 Flume。通常在其配置文件规定 Logstash 如何处理各种类型的事件流,一般包含 input、filter、output 三个部分。Logstash 为各个部分提供相应的插件,因而有 input、filter、output 三类插件完成各种处理和转换;另外 codec 类的插件可以放在 input 和 output 部分通过简单编码来简化处理过程。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">Kibana是ElasticSearch的用户界面。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">在实际应用场景下,为了满足大数据实时检索的场景,利用Filebeat去监控日志文件,将Kafka作为Filebeat的输出端,Kafka实时接收到Filebeat后以Logstash作为输出端输出,到Logstash的数据也许还不是我们想要的格式化或者特定业务的数据,这时可以通过Logstash的一些过了插件对数据进行过滤最后达到想要的数据格式以ElasticSearch作为输出端输出,数据到ElasticSearch就可以进行丰富的分布式检索了。</p> <h2 id="h2" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.4em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">2 下载安装包</span></h2> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">下载 elasticsearch、logstash、kibana、filebeat 的压缩包,并将四个压缩包上传到 /data/elk 目录下<br> </p><figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="http://pp0891ca2.bkt.clouddn.com/Fi7YjNrEE87kB5pam4_So71ux82a" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure><br>官方文档:<br>Filebeat:<br>https://www.elastic.co/cn/products/beats/filebeat<br>Logstash:<br>https://www.elastic.co/cn/products/logstash<br>Kibana:<br>https://www.elastic.co/cn/products/kibana<br>Elasticsearch:<br>https://www.elastic.co/cn/products/elasticsearch<br>elasticsearch中文社区:<br>https://elasticsearch.cn/<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"></p> <h2 id="h3elk" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.4em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3 安装ELK</span></h2> <h3 id="h31jdk18" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; border-bottom: 2px solid rgb(239, 112, 96); font-size: 1.3em;"><span style="font-size: inherit; line-height: inherit; margin: 0px; display: inline-block; font-weight: normal; background: rgb(239, 112, 96); color: rgb(255, 255, 255); padding: 3px 10px 1px; border-top-right-radius: 3px; border-top-left-radius: 3px; margin-right: 3px;">3.1 安装jdk1.8</span><span style="display: inline-block; vertical-align: bottom; border-bottom: 36px solid rgb(239, 235, 233); border-right: 20px solid transparent;"> </span></h3> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">略</p> <h3 id="h32elasticsearch" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; border-bottom: 2px solid rgb(239, 112, 96); font-size: 1.3em;"><span style="font-size: inherit; line-height: inherit; margin: 0px; display: inline-block; font-weight: normal; background: rgb(239, 112, 96); color: rgb(255, 255, 255); padding: 3px 10px 1px; border-top-right-radius: 3px; border-top-left-radius: 3px; margin-right: 3px;">3.2 安装ElasticSearch</span><span style="display: inline-block; vertical-align: bottom; border-bottom: 36px solid rgb(239, 235, 233); border-right: 20px solid transparent;"> </span></h3> <h4 id="h321es" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.2.1 安装配置es</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">说明:ElasticSearch的运行不能用root执行,自己用useradd命令新建一个用户如下所示:<br><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">添加普通用户elk并设置密码</strong><br>useradd elk<br>passwd elk 然后根据提示输入密码即可</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">修改文件所有者</strong><br>chown -R elk:elk /data/elk</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">解压</strong><br>tar -zxvf elasticsearch-6.7.0.tar.gz</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">修改 es 的配置文件</strong><br>cd /data/elk/elasticsearch-6.7.0/config<br>vim elasticsearch.yml<br>修改方法参考如下:</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">cluster.name:&nbsp;my-application<br>node.name:&nbsp;node-1<br>node.attr.rack:&nbsp;r1<br>path.data:&nbsp;/data/elk/elasticsearch-6.7.0/data<br>path.logs:&nbsp;/data/elk/elasticsearch-6.7.0/logs<br>network.host:&nbsp;localhost<br>http.port:&nbsp;9200<br></code></pre> <h4 id="h322" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.2.2 启动</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">su elk<br>./bin/elasticsearch -d</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">启动时间有点慢,耐心等待10-20s或者更长,查看9200,9300端口是否开启 </p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">netstat&nbsp;-tnlp|grep&nbsp;9[23]00<br><br>tcp6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;localhost:9200&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;:::*&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;LISTEN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;30155/java&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>tcp6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;localhost:9300&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;:::*&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;LISTEN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;30155/java<br></code></pre> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">访问地址:http://localhost:9200 <br></p><figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="http://pp0891ca2.bkt.clouddn.com/FlH7Altb51w74YeJb9xirr-uwBzV" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure><br>停止服务:<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"></p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">ps&nbsp;-ef&nbsp;|grep&nbsp;elasticsearch&nbsp;<br>kill&nbsp;PID<br></code></pre> <h3 id="h33filebeat" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; border-bottom: 2px solid rgb(239, 112, 96); font-size: 1.3em;"><span style="font-size: inherit; line-height: inherit; margin: 0px; display: inline-block; font-weight: normal; background: rgb(239, 112, 96); color: rgb(255, 255, 255); padding: 3px 10px 1px; border-top-right-radius: 3px; border-top-left-radius: 3px; margin-right: 3px;">3.3 安装filebeat</span><span style="display: inline-block; vertical-align: bottom; border-bottom: 36px solid rgb(239, 235, 233); border-right: 20px solid transparent;"> </span></h3> <h4 id="h331" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.3.1 安装配置</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">filebeat:部署在具体的业务机器上,通过定时监控的方式获取增量的日志,并转发到logstash、elasticsearch、kafka等等。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">解压</strong><br>tar –zxvf filebeat-6.7.0-linux-x86_64.tar.gz<br>cd filebeat-6.7.0-linux-x86_64<br><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">修改kibana 的配置文件</strong><br>vim filebeat.yml</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">===================Filebeat&nbsp;prospectors&nbsp;===========<br><br>filebeat.prospectors:<br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;"><br>#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">&nbsp;Each&nbsp;-&nbsp;is&nbsp;a&nbsp;prospector.&nbsp;Most&nbsp;options&nbsp;can&nbsp;be&nbsp;<span class="hljs-built_in" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(248, 35, 117); word-wrap: inherit !important; word-break: inherit !important;">set</span>&nbsp;at&nbsp;the&nbsp;prospector&nbsp;level,&nbsp;so</span><br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;">#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">&nbsp;you&nbsp;can&nbsp;use&nbsp;different&nbsp;prospectors&nbsp;<span class="hljs-keyword" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(248, 35, 117); word-wrap: inherit !important; word-break: inherit !important;">for</span>&nbsp;various&nbsp;configurations.</span><br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;">#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">&nbsp;Below&nbsp;are&nbsp;the&nbsp;prospector&nbsp;specific&nbsp;configurations.</span><br><br>-&nbsp;input_type:&nbsp;log<br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;"><br>&nbsp;&nbsp;#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">&nbsp;Paths&nbsp;that&nbsp;should&nbsp;be&nbsp;crawled&nbsp;and&nbsp;fetched.&nbsp;Glob&nbsp;based&nbsp;paths.</span><br>&nbsp;&nbsp;paths:<br>&nbsp;&nbsp;&nbsp;&nbsp;-&nbsp;/data/elk/logfile<br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;">#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">-&nbsp;c:\programdata\elasticsearch\logs\*</span><br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;"><br>#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">输出到logstash</span><br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;">#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">--------------------&nbsp;Logstash&nbsp;output&nbsp;--------------------------------</span><br>output.logstash:<br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;">&nbsp;&nbsp;#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">&nbsp;The&nbsp;Logstash&nbsp;hosts</span><br>&nbsp;&nbsp;hosts:&nbsp;["localhost:5044"]<br></code></pre> <h4 id="h332" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.3.2 启动</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">./filebeat -e -c filebeat.yml -d "publish"</p> <h3 id="h34logstash" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; border-bottom: 2px solid rgb(239, 112, 96); font-size: 1.3em;"><span style="font-size: inherit; line-height: inherit; margin: 0px; display: inline-block; font-weight: normal; background: rgb(239, 112, 96); color: rgb(255, 255, 255); padding: 3px 10px 1px; border-top-right-radius: 3px; border-top-left-radius: 3px; margin-right: 3px;">3.4 安装logstash</span><span style="display: inline-block; vertical-align: bottom; border-bottom: 36px solid rgb(239, 235, 233); border-right: 20px solid transparent;"> </span></h3> <h4 id="h341" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.4.1 安装配置</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">解压</strong><br>tar -zxvf logstash-6.7.0.tar.gz<br>cd logstash-6.7.0<br><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">配置</strong><br>在logstash文件夹的下bin目录创建配置文件logstash.conf ,内容如下:<br>input:对应输入的配置,其中path是监控的文**件路劲, codec编码个格式<br>output:elasticsearch : { hosts =&gt; "localhost:9200" } <br>对应输出到elasticsearch,hosts是elasticsearch安装地址</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">input&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;beats&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;host&nbsp;=&gt;&nbsp;"localhost"<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;port&nbsp;=&gt;&nbsp;5044&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;codec&nbsp;=&gt;&nbsp;json<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br>}<br><br>output&nbsp;{<br><br>&nbsp;&nbsp;&nbsp;&nbsp;stdout&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;codec&nbsp;=&gt;&nbsp;rubydebug<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br><br>&nbsp;&nbsp;&nbsp;&nbsp;elasticsearch&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;hosts&nbsp;=&gt;&nbsp;"localhost:9200"<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;index="test_index"<br>&nbsp;&nbsp;&nbsp;&nbsp;}<br>}<br></code></pre> <h4 id="h342" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.4.2 启动</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">测试你的配置文件 是否正确( 解析配置文件并报告任何错误。)<br>./logstash -f logstash.conf --config.test_and_exit</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">Sending&nbsp;Logstash&nbsp;logs&nbsp;to&nbsp;/data/elk/logstash-6.7.0/logs&nbsp;which&nbsp;is&nbsp;now&nbsp;configured&nbsp;via&nbsp;log4j2.properties<br>[2019-03-28T17:55:34,114][INFO&nbsp;][logstash.setting.writabledirectory]&nbsp;Creating&nbsp;directory&nbsp;{:setting=&gt;"path.queue",&nbsp;:path=&gt;"/data/elk/logstash-6.7.0/data/queue"}<br>[2019-03-28T17:55:34,169][INFO&nbsp;][logstash.setting.writabledirectory]&nbsp;Creating&nbsp;directory&nbsp;{:setting=&gt;"path.dead_letter_queue",&nbsp;:path=&gt;"/data/elk/logstash-6.7.0/data/dead_letter_queue"}<br>[2019-03-28T17:55:34,856][WARN&nbsp;][logstash.config.source.multilocal]&nbsp;Ignoring&nbsp;the&nbsp;'pipelines.yml'&nbsp;file&nbsp;because&nbsp;modules&nbsp;or&nbsp;command&nbsp;line&nbsp;options&nbsp;are&nbsp;specified<br>Configuration&nbsp;OK<br>[2019-03-28T17:55:43,327][INFO&nbsp;][logstash.runner&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;]&nbsp;Using&nbsp;config.test_and_exit&nbsp;mode.&nbsp;Config&nbsp;Validation&nbsp;Result:&nbsp;OK.&nbsp;Exiting&nbsp;Logstash<br></code></pre> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">启动命令(启用自动配置重新加载,这样就不必每次修改配置文件时都停止并重新启动Logstash )<br>./logstash -f logstash.conf --config.reload.automatic</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">停止服务<br>ps aux | grep logstash<br>kill -9 pid</p> <h3 id="h35kibana" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; border-bottom: 2px solid rgb(239, 112, 96); font-size: 1.3em;"><span style="font-size: inherit; line-height: inherit; margin: 0px; display: inline-block; font-weight: normal; background: rgb(239, 112, 96); color: rgb(255, 255, 255); padding: 3px 10px 1px; border-top-right-radius: 3px; border-top-left-radius: 3px; margin-right: 3px;">3.5 安装kibana</span><span style="display: inline-block; vertical-align: bottom; border-bottom: 36px solid rgb(239, 235, 233); border-right: 20px solid transparent;"> </span></h3> <h4 id="h351" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.5.1 安装配置</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">解压</strong><br>tar –zxvf kibana-6.7.0-linux-x86_64.tar.gz<br>cd kibana-6.7.0-linux-x86_64/config<br><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">配置</strong><br>vim kibana.yml</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">server.port:&nbsp;5601<br>server.host:&nbsp;"localhost"<br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;"><br>#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">elasticsearch.username:&nbsp;<span class="hljs-string" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(238, 220, 112); word-wrap: inherit !important; word-break: inherit !important;">"elastic"</span></span><br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;">#</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;">elasticsearch.password:&nbsp;<span class="hljs-string" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(238, 220, 112); word-wrap: inherit !important; word-break: inherit !important;">"changeme"</span></span><br><br>elasticsearch.hosts:&nbsp;["http://localhost:9200"]<br>kibana.index:&nbsp;".kibana"<br></code></pre> <h4 id="h352" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">3.5.2 启动</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">./bin/kibana</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">访问地址:<br>http://localhost:5601 <br> </p><figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="http://pp0891ca2.bkt.clouddn.com/FgCmTbW8llYkiS4yYCzz-DCYnl_6" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">netstat -anltp|grep 5601查找对应端口<br>kill -9 pid</p> <h2 id="h4" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.4em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">4 测试</span></h2> <h3 id="h41kibana" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; border-bottom: 2px solid rgb(239, 112, 96); font-size: 1.3em;"><span style="font-size: inherit; line-height: inherit; margin: 0px; display: inline-block; font-weight: normal; background: rgb(239, 112, 96); color: rgb(255, 255, 255); padding: 3px 10px 1px; border-top-right-radius: 3px; border-top-left-radius: 3px; margin-right: 3px;">4.1 监测指定文件,Kibana展示</span><span style="display: inline-block; vertical-align: bottom; border-bottom: 36px solid rgb(239, 235, 233); border-right: 20px solid transparent;"> </span></h3> <h4 id="h411logstash" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">4.1.1 Logstash监控文件</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">我们通过echo往这文件/data/elk/test.log追加内容,来测试整个日志收集系统是否可行<br>vim test2.conf</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">input&nbsp;{<br>&nbsp;&nbsp;file&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;=&gt;"syslog"<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;path&nbsp;=&gt;&nbsp;["/data/elk/test.log"&nbsp;]<br>&nbsp;&nbsp;}<br>&nbsp;&nbsp;syslog&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;type&nbsp;=&gt;"syslog"<br>&nbsp;&nbsp;&nbsp;&nbsp;port&nbsp;=&gt;"5544"<br>&nbsp;&nbsp;}<br>}<br>output&nbsp;{<br>&nbsp;&nbsp;stdout&nbsp;{&nbsp;codec=&gt;&nbsp;rubydebug&nbsp;}<br>&nbsp;&nbsp;elasticsearch&nbsp;{hosts&nbsp;=&gt;&nbsp;"http://&nbsp;localhost:9200"}<br>}<br></code></pre> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">启动<br>./logstash -f test2.conf --config.reload.automatic</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">向监控文件中写入数据</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">echo&nbsp;"{&nbsp;"firstName":&nbsp;"1",&nbsp;"lastName":"McLaughlin",&nbsp;"email":&nbsp;"aaaa"&nbsp;}"&nbsp;&gt;&gt;&nbsp;test.log&nbsp;<br></code></pre> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">Logstash打印接收到的数据:</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">{<br>&nbsp;&nbsp;&nbsp;&nbsp;"message"&nbsp;=&gt;&nbsp;"{&nbsp;firstName:&nbsp;1,&nbsp;lastName:McLaughlin,&nbsp;email:&nbsp;aaaa&nbsp;}",<br>&nbsp;&nbsp;&nbsp;&nbsp;"@timestamp"&nbsp;=&gt;&nbsp;2019-03-29T02:39:35.229Z,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"@version"&nbsp;=&gt;&nbsp;"1",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"host"&nbsp;=&gt;&nbsp;"node223",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"path"&nbsp;=&gt;&nbsp;"/data/elk/test.log"<br>}<br></code></pre> <h4 id="h412es" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">4.1.2 es 查看接收数据</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"><strong style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; font-weight: bold; color: rgb(233, 105, 0);">查看es所有index</strong><br>curl -X GET 'http://localhost:9200/_cat/indices?v'</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">health&nbsp;status&nbsp;index&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;uuid&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pri&nbsp;rep&nbsp;docs.count&nbsp;docs.deleted&nbsp;store.size&nbsp;pri.store.size<br>green&nbsp;&nbsp;open&nbsp;&nbsp;&nbsp;.kibana_task_manager&nbsp;ZYN7zZl3QLuB1d-EE_ysHA&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;12.5kb&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;12.5kb<br>yellow&nbsp;open&nbsp;&nbsp;&nbsp;logstash-2019.03.29&nbsp;&nbsp;rfkkxW7rQDKnl40_Ekaf7g&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10.8kb&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10.8kb<br>yellow&nbsp;open&nbsp;&nbsp;&nbsp;test-index&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;hFBQF-jZSkqajPRYzegfwg&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1.2kb&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1.2kb<br>green&nbsp;&nbsp;open&nbsp;&nbsp;&nbsp;.kibana_1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sEUUnVMxQ0amHbQGcOedOQ&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;19.8kb&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;19.8kb<br></code></pre> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">查看单个index数据<br>curl 'localhost:9200/logstash-2019.03.29/_search?pretty=true'</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">{<br>&nbsp;&nbsp;"took"&nbsp;:&nbsp;4,<br>&nbsp;&nbsp;"timed_out"&nbsp;:&nbsp;false,<br>&nbsp;&nbsp;"_shards"&nbsp;:&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;"total"&nbsp;:&nbsp;5,<br>&nbsp;&nbsp;&nbsp;&nbsp;"successful"&nbsp;:&nbsp;5,<br>&nbsp;&nbsp;&nbsp;&nbsp;"skipped"&nbsp;:&nbsp;0,<br>&nbsp;&nbsp;&nbsp;&nbsp;"failed"&nbsp;:&nbsp;0<br>&nbsp;&nbsp;},<br>&nbsp;&nbsp;"hits"&nbsp;:&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;"total"&nbsp;:&nbsp;2,<br>&nbsp;&nbsp;&nbsp;&nbsp;"max_score"&nbsp;:&nbsp;1.0,<br>&nbsp;&nbsp;&nbsp;&nbsp;"hits"&nbsp;:&nbsp;[<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"_index"&nbsp;:&nbsp;"logstash-2019.03.29",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"_type"&nbsp;:&nbsp;"doc",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"_id"&nbsp;:&nbsp;"W85Rx2kBAxlZ_OvPIlZf",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"_score"&nbsp;:&nbsp;1.0,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"_source"&nbsp;:&nbsp;{<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"message"&nbsp;:&nbsp;"{&nbsp;firstName:&nbsp;1,&nbsp;lastName:McLaughlin,&nbsp;email:&nbsp;aaaa&nbsp;}",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"@timestamp"&nbsp;:&nbsp;"2019-03-29T02:39:35.229Z",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"@version"&nbsp;:&nbsp;"1",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"host"&nbsp;:&nbsp;"node223",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"path"&nbsp;:&nbsp;"/data/elk/test.log"<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<br>&nbsp;&nbsp;&nbsp;&nbsp;]<br>&nbsp;&nbsp;}<br>}<br></code></pre> <h4 id="h413eskibana" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">4.1.3 用ES连接到Kibana</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">在你开始用Kibana之前,你需要告诉Kibana你想探索哪个Elasticsearch索引。第一次访问Kibana是,系统会提示你定义一个索引模式以匹配一个或多个索引的名字。</p> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">1、访问Kibana UI。例如,localhost:56011 或者 http://YOURDOMAIN.com:5601<br>2、指定一个索引模式来匹配一个或多个你的Elasticsearch索引。当你指定了你的索引模式以后,任何匹配到的索引都将被展示出来。<br>  (画外音:*匹配0个或多个字符; 指定索引默认是为了匹配索引,确切的说是匹配索引名字)<br>3、点击“Next Step”以选择你想要用来执行基于时间比较的包含timestamp字段的索引。如果你的索引没有基于时间的数据,那么选择“I don’t want to use the Time Filter”选项。<br>4、点击“Create index pattern”按钮来添加索引模式。第一个索引模式自动配置为默认的索引默认,以后当你有多个索引模式的时候,你就可以选择将哪一个设为默认。(提示:Management &gt; Index Patterns)<br></p><figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="http://pp0891ca2.bkt.clouddn.com/Fl3IvJMPoQ09oO3qAye7cN_5cPW9" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure><br><figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="http://pp0891ca2.bkt.clouddn.com/FhRNAU4k6aH7qJRP7DxFvs5Bk8-S" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">现在,Kibana已经连接到你的Elasticsearch数据。Kibana展示了一个只读的字段列表,这些字段是匹配到的这个索引配置的字段。</p> <h4 id="h414kibana" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">4.1.4 Kibana展示</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">http://localhost:5601</p> <h3 id="h42springboot" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; border-bottom: 2px solid rgb(239, 112, 96); font-size: 1.3em;"><span style="font-size: inherit; line-height: inherit; margin: 0px; display: inline-block; font-weight: normal; background: rgb(239, 112, 96); color: rgb(255, 255, 255); padding: 3px 10px 1px; border-top-right-radius: 3px; border-top-left-radius: 3px; margin-right: 3px;">4.2 监控Spring Boot服务日志</span><span style="display: inline-block; vertical-align: bottom; border-bottom: 36px solid rgb(239, 235, 233); border-right: 20px solid transparent;"> </span></h3> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">要求日志格式为json格式</p> <h4 id="h421json" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">4.2.1 json日志格式配置</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">在Spring Boot工程resources目录下配置logback.xml文件,旨在将log4j日志按json格式输出到指定目录,logback.xml文件内容如下:</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">&lt;?xml&nbsp;version="1.0"&nbsp;encoding="UTF-8"?&gt;<br>&lt;configuration&nbsp;debug="false"&gt;<br>&nbsp;&nbsp;&nbsp;&lt;!--定义日志文件的存储地址&nbsp;勿在&nbsp;LogBack&nbsp;的配置中使用相对路径&nbsp;<br>&nbsp;&nbsp;&nbsp;&lt;property&nbsp;name="LOG_HOME"&nbsp;value="E:/demo/home"&nbsp;/&gt;<br><span class="hljs-meta" style="font-size: inherit; line-height: inherit; margin: 0px; padding: 0px; color: rgb(91, 218, 237); word-wrap: inherit !important; word-break: inherit !important;">&nbsp;&nbsp;&nbsp;--&gt;</span><span class="bash" style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px; word-wrap: inherit !important; word-break: inherit !important;"></span><br><br>&nbsp;&nbsp;&nbsp;&lt;!--格式化输出:%d表示日期,%thread表示线程名,%-5level:级别从左显示5个字符宽度%msg:日志消息,%n是换行符&nbsp;--&gt;<br>&nbsp;&nbsp;&nbsp;&lt;!--&lt;property&nbsp;name="LOG_PATTERN"&nbsp;value="[&nbsp;&nbsp;%d{yyyy-MM-dd&nbsp;HH:mm:ss.SSS}]&nbsp;[UUID:%X{requestId}]&nbsp;[%level]|[${HOSTNAME}]&nbsp;[%thread]|[%logger{36}]&nbsp;|&nbsp;%msg|%n&nbsp;"&nbsp;/&gt;--&gt;<br>&nbsp;&nbsp;&nbsp;&lt;property&nbsp;name="TIME_PATTERN"&nbsp;value="yyyy-MM-dd'T'HH:mm:ss.SSS+08:00"&nbsp;/&gt;<br>&nbsp;&nbsp;&nbsp;&lt;property&nbsp;name="LOG_PATTERN"&nbsp;value='{"event":"log","timestamp":"%d{${TIME_PATTERN}}","level":"%level","serviceName":"${springAppName:-}","pid":&nbsp;"${PID:-}","host":"${HOSTNAME}","class":&nbsp;"%logger{40}","thread":"%thread","message":"%msg"}%n&nbsp;'&nbsp;/&gt;<br><br>&nbsp;&nbsp;&nbsp;&lt;!--&nbsp;Console&nbsp;输出设置&nbsp;--&gt;<br>&nbsp;&nbsp;&nbsp;&lt;appender&nbsp;name="STDOUT"&nbsp;class="ch.qos.logback.core.ConsoleAppender"&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;encoder&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;pattern&gt;${LOG_PATTERN}&lt;/pattern&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;charset&gt;utf8&lt;/charset&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;/encoder&gt;<br>&nbsp;&nbsp;&nbsp;&lt;/appender&gt;<br><br>&nbsp;&nbsp;&nbsp;&lt;!--&nbsp;按照每天生成日志文件&nbsp;--&gt;<br>&nbsp;&nbsp;&nbsp;&lt;appender&nbsp;name="FILE"&nbsp;class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;file&gt;logs/newsinfo-dataservice.log&lt;/file&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;rollingPolicy&nbsp;class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;FileNamePattern&gt;logs/newsinfo-dataservice.%d{yyyy-MM-dd}.log&lt;/FileNamePattern&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;/rollingPolicy&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;encoder&nbsp;class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;pattern&gt;${LOG_PATTERN}&lt;/pattern&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;/encoder&gt;<br>&nbsp;&nbsp;&nbsp;&lt;/appender&gt;<br><br>&nbsp;&nbsp;&nbsp;&lt;!--log4jdbc&nbsp;--&gt;<br>&nbsp;&nbsp;&nbsp;&lt;logger&nbsp;name="com.sohu"&nbsp;level="debug"&nbsp;additivity="false"&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;appender-ref&nbsp;ref="STDOUT"&nbsp;/&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;appender-ref&nbsp;ref="FILE"&nbsp;/&gt;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&lt;/logger&gt;<br><br>&nbsp;&nbsp;&nbsp;&lt;!--&nbsp;日志输出级别&nbsp;--&gt;<br>&nbsp;&nbsp;&nbsp;&lt;root&nbsp;level="info"&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;appender-ref&nbsp;ref="STDOUT"&nbsp;/&gt;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;appender-ref&nbsp;ref="FILE"&nbsp;/&gt;<br>&nbsp;&nbsp;&nbsp;&lt;/root&gt;<br>&lt;/configuration&gt;<br></code></pre> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">输出日志示例如下:</p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">{"event":"log","timestamp":"2019-04-09T19:47:07.908Z","level":"INFO","serviceName":"springAppName_IS_UNDEFINED","pid":&nbsp;"7196","host":"node1","class":&nbsp;"o.s.b.web.servlet.FilterRegistrationBean","thread":"localhost-startStop-1","message":"Mapping&nbsp;filter:&nbsp;'MyFilter'&nbsp;to&nbsp;urls:&nbsp;[/getUser,&nbsp;/hello]"}<br></code></pre> <h4 id="h422domeosspringboot" style="color: inherit; line-height: inherit; padding: 0px; margin: 1.6em 0px; font-weight: bold; font-size: 1.2em;"><span style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;">4.2.2 DomeOS启动Spring Boot服务</span></h4> <p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;">启动服务的同时,配置日志收集到kafka,然后调用ELK读取kafka日志存入ES,进行日志的全文检索和实时告警分析。<br></p><figure style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><img src="http://pp0891ca2.bkt.clouddn.com/Fr9j5kVPu311qkX6nTm7yle-J42H" alt="" title="" style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; display: block; margin: 0px auto; max-width: 100%;"><figcaption style="line-height: inherit; margin: 0px; padding: 0px; margin-top: 10px; text-align: center; color: rgb(153, 153, 153); font-size: 0.7em;"></figcaption></figure><br>创建新版本时,如上配置下日志收集模块,日志就自动通过Flume接入Kafka了。<p style="font-size: inherit; color: inherit; line-height: inherit; padding: 0px; margin: 1.7em 0px;"></p> <pre style="font-size: inherit; color: inherit; line-height: inherit; margin: 0px; padding: 0px;"><code class="shell language-shell hljs" style="overflow-wrap: break-word; margin: 0px 2px; line-height: 18px; font-size: 14px; font-weight: normal; word-spacing: 0px; letter-spacing: 0px; font-family: Consolas, Inconsolata, Courier, monospace; border-radius: 0px; color: rgb(169, 183, 198); background: rgb(40, 43, 46); overflow-x: auto; padding: 0.5em; display: block !important; white-space: pre !important; word-wrap: normal !important; word-break: normal !important; overflow: auto !important;">路径:/code/logs/newsinfo-dataservice.log<br>自动收集&nbsp;:自动将日志收集到&nbsp;Kafka<br>Kafka&nbsp;Topic:newsinfo_appdata_service_log<br></code></pre></div>

o
粉丝 0
博文 500
码字总数 0
作品 0
私信 提问
加载中
请先登录后再评论。
记一次失败的Perl + Nginx + FastCGI 配置过程

这两天心血来潮,不知道为什么和 Perl + Nginx + FastCGI 配置 耗上了。但是失败了,记录如下: 1)安装Nginx 1.4.3 ,我的是WINDOWS 7 系统,修改配置文件如下: location ~ .(pl|cgi|perl)?...

通吃岛-低手哥
2013/10/27
1.7K
7
访问安全控制解决方案

本文是《轻量级 Java Web 框架架构设计》的系列博文。 今天想和大家简单的分享一下,在 Smart 中是如何做到访问安全控制的。也就是说,当没有登录或 Session 过期时所做的操作,会自动退回到...

黄勇
2013/11/03
3.5K
8
Flappy Bird(安卓版)逆向分析(一)

更改每过一关的增长分数 反编译的步骤就不介绍了,我们直接来看反编译得到的文件夹 方法1:在smali目录下,我们看到org/andengine/,可以知晓游戏是由andengine引擎开发的。打开/res/raw/at...

enimey
2014/03/04
6.1K
18
我的架构演化笔记 功能1: 基本的用户注册

“咚咚”,一阵急促的敲门声, 我从睡梦中惊醒,我靠,这才几点,谁这么早, 开门一看,原来我的小表弟放暑假了,来南京玩,顺便说跟我后面学习一个网站是怎么做出来的。 于是有了下面的一段...

强子哥哥
2014/05/31
976
3
Swift百万线程攻破单例(Singleton)模式

一、不安全的单例实现 在上一篇文章我们给出了单例的设计模式,直接给出了线程安全的实现方法。单例的实现有多种方法,如下面: class SwiftSingleton { } 这段代码的实现,在shared中进行条...

一叶博客
2014/06/20
3.4K
16

没有更多内容

加载失败,请刷新页面

加载更多

SQL 语句大全

点击上方“掌上编程”,选择“置顶或者星标” 优质文章第一时间送达! 一、基础 「1、说明:创建数据库」 CREATE DATABASE database-name    「2、说明:删除数据库」 drop database ...

GeneralMa
昨天
0
0
山东创睦网络科技有限公司:使用Python爬取全球新冠肺炎疫情数据

使用Python爬取全球新冠肺炎疫情数据 导入所需库包 获取实时数据的url 正式编写程序 查看输出结果 导入所需库包 在获取数据之前,我们需要先安装好所需的包requests和pandas: 1.如果是使用p...

osc_qv1fwke0
34分钟前
14
0
如何1年获得别人3年的工作经验(深度好文)

最近有同学问我,为什么你的工作年限不长,技术却这么厉害,我笑了笑,啥也没说。 我不是不想回答,是不知道怎么回答。在他们的定位可能就是,每方面都懂一点,遇到问题能够快速解决,就是比...

zhang_rick
今天
1
0
新基建带动行业

什么是“新基建”? 什么是“新基建”? 根据央视发布的信息来看,其涵盖了5G基站建设、新能源汽车充电桩、大数据中心、人工智能、工业互联网,特高压,城际以及城轨交通,涉及了七大领域和相...

osc_anefoz50
34分钟前
8
0
怕入错行?这群技术人写了本“择业指南”

计算机专业好找工作吗?哪些方向是当前的主流和热门方向呢? 计算机专业的你是不是还在为职业发展纠结犹豫呢? 刚经历完高考选专业的你是不是还在迷茫徘徊呢? 那么福利来啦! 《软件技术职业...

阿里云云栖号
34分钟前
21
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部