文档章节

flume 1.7

柯里昂
 柯里昂
发布于 2017/09/07 17:45
字数 486
阅读 17
收藏 0
# 官网 简单的入门例子
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
# 启动 flume agent
bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
# 同节点,另起一个终端
nc localhost 44444
# 同时输出到终端和文件
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k2 k1
a1.channels = c1 c2

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = dn7
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k2.type = file_roll
a1.sinks.k2.sink.directory=/tmp/k2/

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Use a channel which buffers events in memory
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 c2

a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
# 同时在控制台,hdfs,file-roll
# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k2 k1 k3
a1.channels = c1 c2 c3

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = dn6
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

a1.sinks.k2.type = file_roll
a1.sinks.k2.sink.directory=/tmp/k2/

a1.sinks.k3.type=hdfs
a1.sinks.k3.hdfs.path=hdfs://nn2:8020/tmp/test/flume/
a1.sinks.k3.hdfs.filePrefix=test-
a1.sinks.k3.hdfs.round=true
a1.sinks.k3.hdfs.roundValue=10
a1.sinks.k3.hdfs.roundUnit=second
a1.sinks.k3.hdfs.fileType=DataStream

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Use a channel which buffers events in memory
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

# Use a channel which buffers events in memory
a1.channels.c3.type = memory
a1.channels.c3.capacity = 1000
a1.channels.c3.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1 c2 c3

a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
a1.sinks.k3.channel = c3
# 将agent的配置文件上传到 zk后,所有运行agent的节点都可从zk获取
bin/flume-ng agent –conf conf -z zkhost:2181,zkhost1:2181 -p /flume –name a1 -Dflume.root.logger=INFO,console

第三方插件

Flume Channels: Memory Channel

Flume Sinks: logger

# -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true

© 著作权归作者所有

共有 人打赏支持
上一篇: 比较实用的博文
下一篇: linux 网卡
柯里昂
粉丝 25
博文 174
码字总数 82846
作品 0
徐汇
技术主管
私信 提问
Apache Flume 1.5.2 发布,日志服务器

Apache Flume 1.5.2 发布,此版本现已提供下载:http://flume.apache.org/download.html。 主要更新内容如下: [FLUME-2547] - Removing SSLv2Hello causes Java 6 clients to break [FLUME......

oschina
2014/11/20
2.8K
5
Apache Flume 1.7.0 发布,日志服务器

Apache Flume 1.7.0 发布了,Flume 是一个分布式、可靠和高可用的服务,用于收集、聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型。这是一个可靠、容错的服务。 本次更新如...

局长
2016/10/19
2K
3
flume 1.7 源码导入eclipse windows

安装maven,设置MAVEN_HOME等配置 下载flume源码 eclipse-oxygen,设置eclipse 使用外部maven,并配置settings.xml 遇到问题: 如果顺利,已将所需jar都下载下来了。 导入后遇到如下问题 fl...

柯里昂
2017/10/31
0
0
Apache Flume 1.6.0 发布,日志服务器

Apache Flume 1.6.0 发布,此版本现已提供下载: http://flume.apache.org/download.html 更新内容: ** Bug 修复 [FLUME-1793] - Unit test TestElasticSearchLogStashEventSerializer fail......

oschina
2015/06/03
3.1K
2
Apache Flume 1.5.0 发布,日志服务器

Apache Flume 1.5.0 发布,Flume 是一个分布式、可靠和高可用的服务,用于收集、聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型。这是一个可靠、容错的服务。 改进内容包括...

oschina
2014/05/22
2.8K
4

没有更多内容

加载失败,请刷新页面

加载更多

从 JVM 分析 hibernate-validator NoClassDefFoundError

最近排查一个spring boot应用抛出hibernate.validator NoClassDefFoundError的问题,异常信息如下: Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.hibernat......

微笑向暖wx
15分钟前
0
0
c++指针和字符串

天王盖地虎626
19分钟前
0
0
从 JVM 分析 hibernate-validator NoClassDefFoundError

最近排查一个spring boot应用抛出hibernate.validator NoClassDefFoundError的问题,异常信息如下: Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.hibernat......

java菜分享
24分钟前
1
0
超500款社交APP对标微信,细分未来社交领域

“微信之父”张小龙在2019微信公开课PRO活动上透露,截止2018年8月,微信的日登录量已超过10亿。腾讯2018年三季度财报显示,QQ智能终端月活跃账户达6.98亿人。 当前微信和QQ无疑是中国最大且...

ThinkSNS账号
31分钟前
1
0
Fiddler 抓包工具总结

序章 Fiddler是一个蛮好用的抓包工具,可以将网络传输发送与接受的数据包进行截获、重发、编辑、转存等操作。也可以用来检测网络安全。反正好处多多,举之不尽呀!当年学习的时候也蛮费劲,一...

javaer
33分钟前
1
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部