flume监控一个linux指定的一个文件夹的文件信息

小程序：扫一扫查出行
【扫一扫了解最新限行尾号】
复制小程序

1.编辑一个配置文件 flume-app.conf 拷贝至fulme的安装目录的conf下

# The configuration file needs to define the sources,
# the channels and the sinks. ### define agent
a3.sources = r3
a3.channels = c3
a3.sinks = k3 ### define sources
a3.sources.r3.type = spooldir
### 要扫描的文件夹
a3.sources.r3.spoolDir = /opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6/spoollogs
### 以.log结尾的文件不扫描
a3.sources.r3.ignorePattern = ^(.)*\\.log$
### 扫描完成的文件加一个后缀
a3.sources.r3.fileSuffix = .delete ### define channels
a3.channels.c3.type = file
a3.channels.c3.checkpointDir = /opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6/filechannel/checkpoint
a3.channels.c3.dataDirs = /opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6/filechannel/data ### define sink
a3.sinks.k3.type = hdfs
### 已当天日期在hdfs上创建一个文件夹
a3.sinks.k3.hdfs.path = hdfs://hadoop:8020/user/xuyou/flume/splogs/%Y%m%d
a3.sinks.k3.hdfs.fileType = DataStream
a3.sinks.k3.hdfs.writeFormat = Text
a3.sinks.k3.hdfs.batchSize = 10
a3.sinks.k3.hdfs.useLocalTimeStamp = true ### bind the soures and sink to the channel
a3.sources.r3.channels = c3
a3.sinks.k3.channel = c3