文档章节

flume 1.7

柯里昂
 柯里昂
发布于 2017/09/07 17:45
字数 486
阅读 16
收藏 0
# 官网 简单的入门例子
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
# 启动 flume agent
bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
# 同节点,另起一个终端
nc localhost 44444
# 同时输出到终端和文件
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k2 k1
a1.channels = c1 c2

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = dn7
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k2.type = file_roll
a1.sinks.k2.sink.directory=/tmp/k2/

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Use a channel which buffers events in memory
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 c2

a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
# 同时在控制台,hdfs,file-roll
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k2 k1 k3
a1.channels = c1 c2 c3

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = dn6
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

a1.sinks.k2.type = file_roll
a1.sinks.k2.sink.directory=/tmp/k2/

a1.sinks.k3.type=hdfs
a1.sinks.k3.hdfs.path=hdfs://nn2:8020/tmp/test/flume/
a1.sinks.k3.hdfs.filePrefix=test-
a1.sinks.k3.hdfs.round=true
a1.sinks.k3.hdfs.roundValue=10
a1.sinks.k3.hdfs.roundUnit=second
a1.sinks.k3.hdfs.fileType=DataStream

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Use a channel which buffers events in memory
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

# Use a channel which buffers events in memory
a1.channels.c3.type = memory
a1.channels.c3.capacity = 1000
a1.channels.c3.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 c2 c3

a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
a1.sinks.k3.channel = c3
# 将agent的配置文件上传到 zk后,所有运行agent的节点都可从zk获取
bin/flume-ng agent –conf conf -z zkhost:2181,zkhost1:2181 -p /flume –name a1 -Dflume.root.logger=INFO,console

第三方插件

Flume Channels: Memory Channel

Flume Sinks: logger

# -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true

© 著作权归作者所有

共有 人打赏支持
柯里昂
粉丝 25
博文 173
码字总数 81568
作品 0
徐汇
技术主管
flume 1.7 源码导入eclipse windows

安装maven,设置MAVEN_HOME等配置 下载flume源码 eclipse-oxygen,设置eclipse 使用外部maven,并配置settings.xml 遇到问题: 如果顺利,已将所需jar都下载下来了。 导入后遇到如下问题 fl...

柯里昂
2017/10/31
0
0
追加内容到文件末尾 - flume-append-file-sink

flume追加内容到文件末尾的sink。 仅适合1g以内的日志,再大估计会有压力,没具体测试。 依赖 Flume-NG >= 1.7 Linux MacOS 构建 $ mvn clean package jar文件下载 flume-append-file-sink-1...

五十风
04/13
0
0
Cloudera Flume简介

Flume是Cloudera提供的日志收集系统,Flume支持在日志系统中定制各类数据发送方,用于收集数据;同时,Flume提供对数据进行简单处理,并写到各种数据接受方(可定制)的能力。 Flume是一个分布式...

Javafans
2012/03/12
0
0
Flume日志收集分层架构应用实践

Flume作为一个日志收集工具,非常轻量级,基于一个个Flume Agent,能够构建一个很复杂很强大的日志收集系统,它的灵活性和优势,主要体现在如下几点: 1)模块化设计:在其Flume Agent内部可...

workming
06/29
0
0
Flume OG和Flume NG的区别

应用场景 Flume作为Hadoop中的日志采集工具,非常的好用,但是在安装Flume的时候,查阅很多资料,发现形形色色,有的说安装Flume很简单,有的说安装Flume很复杂,需要依赖zookeeper,所以一方...

wsc449
2017/11/23
0
0

没有更多内容

加载失败,请刷新页面

加载更多

崩溃bug日志总结1

目录介绍 1.1 java.lang.UnsatisfiedLinkError找不到so库异常 1.2 java.lang.IllegalStateException非法状态异常 1.3 android.content.res.Resources$NotFoundException 1.4 java.lang.Ille......

潇湘剑雨
56分钟前
0
0
学习大数据为什么要先学Java?

计算机编程语言有很多,目前用的多一点的就是Java,C++,Python等等。目前大多数学习大数据的人都是选择学习Java,那Java到底好在哪呢?为什么学大数据之前要先学Java呢?我们今天就来分析一...

董黎明
今天
1
0
php删除服务器所有session

php删除服务器所有session踢掉所有在线用户linux 注意:如果要删除服务器上所有session,重启php服务是解决不了问题的,php的session是持久化的。 有效解决办法: 删除 /tmp 下的所有文件(默...

妖尾巴
今天
0
0
Ubuntu18.04 安装最新版WPS

1.手动卸载libreoffice:sudo apt-get remove --purge libreoffice* 2.官网下载WPS和字体: WPS:http://wps-community.org/download.html 字体:http://wps-community.org/download.html?vl......

AI_SKI
今天
4
0
数据结构(算法)-图(深度优先搜索 DFS)

#include <iostream>using namespace std;#define MaxVex 30typedef char VertexType;typedef struct vexNode adjList[MaxVex];struct edgeNode{int adjvex;//邻接点......

ashuo
今天
1
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部