文档章节

flume channel, source, sink的汇总

p
 piggogo
发布于 2017/02/19 11:05
字数 456
阅读 22
收藏 0

 

 

Component

Type

Description

Implementation Class

Channel

memory

In-memory, fast, non-durable event transport

MemoryChannel

Channel

file

A channel for reading, writing, mapping, and manipulating a file

FileChannel

Channel

jdbc

JDBC-based, durable event transport (Derby-based)

JDBCChannel

Channel

recoverablememory

A durable channel implementation that uses the local file system for its storage

RecoverableMemoryChannel

Channel

org.apache.flume.channel.PseudoTxnMemoryChannel

Mainly for testing purposes. Not meant for production use.

PseudoTxnMemoryChannel

Channel

(custom type as FQCN)

Your own Channel impl.

(custom FQCN)

Source

avro

Avro Netty RPC event source

AvroSource

Source

exec

Execute a long-lived Unix process and read from stdout

ExecSource

Source

netcat

Netcat style TCP event source

NetcatSource

Source

seq

Monotonically incrementing sequence generator event source

SequenceGeneratorSource

Source

org.apache.flume.source.StressSource

Mainly for testing purposes. Not meant for production use. Serves as a continuous source of events where each event has the same payload. The payload consists of some number of bytes (specified bysize property, defaults to 500) where each byte has the signed value Byte.MAX_VALUE (0x7F, or 127).

org.apache.flume.source.StressSource

Source

syslogtcp

SyslogTcpSource

Source

syslogudp

SyslogUDPSource

Source

org.apache.flume.source.avroLegacy.AvroLegacySource

AvroLegacySource

Source

org.apache.flume.source.thriftLegacy.ThriftLegacySource

ThriftLegacySource

Source

org.apache.flume.source.scribe.ScribeSource

ScribeSource

Source

(custom type as FQCN)

Your own Source impl.

(custom FQCN)

Sink

hdfs

Writes all events received to HDFS (with support for rolling, bucketing, HDFS-200 append, and more)

HDFSEventSink

Sink

org.apache.flume.sink.hbase.HBaseSink

A simple sink that reads events from a channel and writes them to HBase.

org.apache.flume.sink.hbase.HBaseSink

Sink

org.apache.flume.sink.hbase.AsyncHBaseSink

org.apache.flume.sink.hbase.AsyncHBaseSink

Sink

logger

Log events at INFO level via configured logging subsystem (log4j by default)

LoggerSink

Sink

avro

Sink that invokes a pre-defined Avro protocol method for all events it receives (when paired with an avro source, forms tiered collection)

AvroSink

Sink

file_roll

RollingFileSink

Sink

irc

IRCSink

Sink

null

/dev/null for Flume – blackhole all events received

NullSink

Sink

(custom type as FQCN)

Your own Sink impl.

(custom FQCN)

ChannelSelector

replicating

ReplicatingChannelSelector

ChannelSelector

multiplexing

MultiplexingChannelSelector

ChannelSelector

(custom type)

Your own ChannelSelector impl.

(custom FQCN)

SinkProcessor

default

DefaultSinkProcessor

SinkProcessor

failover

FailoverSinkProcessor

SinkProcessor

load_balance

Provides the ability to load-balance flow over multiple sinks.

LoadBalancingSinkProcessor

SinkProcessor

(custom type as FQCN)

Your own SinkProcessor impl.

(custom FQCN)

Interceptor$Builder

host

HostInterceptor$Builder

Interceptor$Builder

timestamp

TimestampInterceptor

TimestampInterceptor$Builder

Interceptor$Builder

static

StaticInterceptor$Builder

Interceptor$Builder

regex_filter

RegexFilteringInterceptor$Builder

Interceptor$Builder

(custom type as FQCN)

Your own Interceptor$Builder impl.

(custom FQCN)

EventSerializer$Builder

text

BodyTextEventSerializer$Builder

EventSerializer$Builder

avro_event

FlumeEventAvroEventSerializer$Builder

EventSerializer

org.apache.flume.sink.hbase.SimpleHbaseEventSerializer

SimpleHbaseEventSerializer

EventSerializer

org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer

SimpleAsyncHbaseEventSerializer

EventSerializer

org.apache.flume.sink.hbase.RegexHbaseEventSerializer

RegexHbaseEventSerializer

HbaseEventSerializer

Custom implementation of serializer for HBaseSink.
(custom type as FQCN)

Your own HbaseEventSerializer impl.

(custom FQCN)

AsyncHbaseEventSerializer

Custom implementation of serializer for AsyncHbase sink.
(custom type as FQCN)

Your own AsyncHbaseEventSerializer impl.

(custom FQCN)

EventSerializer$Builder

Custom implementation of serializer for all sinks except for HBaseSink and AsyncHBaseSink.
(custom type as FQCN)

Your own EventSerializer$Builder impl.

 

本文转载自:http://abloz.com/2013/02/26/flume-channel-source-sink-summary.html

p
粉丝 1
博文 5
码字总数 3895
作品 0
高级程序员
私信 提问
Apache Flume 1.7.0 发布,日志服务器

Apache Flume 1.7.0 发布了,Flume 是一个分布式、可靠和高可用的服务,用于收集、聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型。这是一个可靠、容错的服务。 本次更新如...

局长
2016/10/19
2.1K
3
Apache Flume 1.5.0 发布,日志服务器

Apache Flume 1.5.0 发布,Flume 是一个分布式、可靠和高可用的服务,用于收集、聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型。这是一个可靠、容错的服务。 改进内容包括...

oschina
2014/05/22
2.8K
4
Apache Flume 1.6.0 发布,日志服务器

Apache Flume 1.6.0 发布,此版本现已提供下载: http://flume.apache.org/download.html 更新内容: ** Bug 修复 [FLUME-1793] - Unit test TestElasticSearchLogStashEventSerializer fail......

oschina
2015/06/03
3.1K
2
02. Spark Streaming实时流处理学习——分布式日志收集框架Flume

2. 分布式日志收集框架Flume 2.1 业务现状分析 如上图,大量的系统和各种服务的日志数据持续生成。用户有了很好的商业创意想要充分利用这些系统日志信息。比如用户行为分析,轨迹跟踪等等。 ...

牦牛sheriff
2018/09/02
0
0
Apache Flume 1.9.0 发布,日志服务器

Apache Flume 1.9.0 发布了,Flume 是一个分布式、可靠和高可用的服务,用于收集、聚合以及移动大量日志数据,使用一个简单灵活的架构,就流数据模型。这是一个可靠、容错的服务。 本次更新如...

程六金
01/10
1K
4

没有更多内容

加载失败,请刷新页面

加载更多

《Designing.Data-Intensive.Applications》笔记 四

第九章 一致性与共识 分布式系统最重要的的抽象之一是共识(consensus):让所有的节点对某件事达成一致。 最终一致性(eventual consistency)只提供较弱的保证,需要探索更高的一致性保证(stro...

丰田破产标志
今天
6
0
docker 使用mysql

1, 进入容器 比如 myslq1 里面进行操作 docker exec -it mysql1 /bin/bash 2. 退出 容器 交互: exit 3. mysql 启动在容器里面,并且 可以本地连接mysql docker run --name mysql1 --env MY...

之渊
今天
7
0
python数据结构

1、字符串及其方法(案例来自Python-100-Days) def main(): str1 = 'hello, world!' # 通过len函数计算字符串的长度 print(len(str1)) # 13 # 获得字符串首字母大写的...

huijue
今天
5
0
OSChina 周日乱弹 —— 我,小小编辑,食人族酋长

Osc乱弹歌单(2019)请戳(这里) 【今日歌曲】 @宇辰OSC :分享娃娃的单曲《飘洋过海来看你》: #今日歌曲推荐# 《飘洋过海来看你》- 娃娃 手机党少年们想听歌,请使劲儿戳(这里) @宇辰OSC...

小小编辑
今天
1K
11
MongoDB系列-- SpringBoot 中对 MongoDB 的 基本操作

SpringBoot 中对 MongoDB 的 基本操作 Database 库的创建 首先 在MongoDB 操作客户端 Robo 3T 中 创建数据库: 增加用户User: 创建 Collections 集合(类似mysql 中的 表): 后面我们大部分都...

TcWong
今天
40
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部