[Flume] 大数据开源环境搭建(集群):30.Flume

常用的华为FusionInsight C60U10中各组件的版本,以此作为兼容参考:

HDFS:2.7.2
Hive:1.3.0
HBase:1.0.2
Spark:1.5.1
Solr:5.3.1
Flume:1.6.0
Kafka:2.10-0.10.0.0
Storm:0.10.0
Hue:3.9.0
Redis:3.0.5
Oozie:4.2.0
Sqoop-1.99.3
其他:MetaData:0.0.1 KrbServer:1.10.7 LdapServer:1.0.0 FTP-Server:0.0.1

本文配置:Redhat6.5、JDK-jdk1.7.0_79 、flume-1.7.0

详细步骤如下:

1.下载Flume http://www.apache.org/dyn/closer.lua/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
我的计划安装目录: /home/hadoop/BigData/flume-1.7.0-bin

$ tar zxvf apache-flume-1.7.0-bin.tar.gz -C /home/hadoop/BigData/
$ mv /home/hadoop/BigData/apache-flume-1.7.0-bin /home/hadoop/BigData/flume-1.7.0-bin

 

2.配置flume-env.sh,仅需要配置JAVA_HOME即可参考如下:

export JAVA_HOME=/usr/local/jdk1.7.0_79

 

3.使用Flume。这里举一个tail -f日志文件fileName.log,并输出到指定目录Sink01/的例子
Step3.1 创建agent文件,比如 vi /home/hadoop/BigData/flume-1.7.0-bin/test_agent/agent_tailF2File.conf 内容如下;

a1.sources = tailF2File
a1.channels = file_channel
a1.sinks = k1

#sources Config
a1.sources.tailF2File.type = exec
a1.sources.tailF2File.command = tail -F /home/hadoop/BigData/flume-1.7.0-bin/test/fileName.log

#channels Config
a1.channels.file_channel.type = memory
a1.channels.file_channel.capacity = 1000000
a1.channels.file_channel.transactionCapacity = 1000

#sinks Config
a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.directory = /home/hadoop/BigData/flume-1.7.0-bin/test/Sink01/
a1.sinks.k1.rollInterval = 0

#bind Config
a1.sources.tailF2File.channels = file_channel
a1.sinks.k1.channel = file_channel

 

Step3.2 启动flume

$ /home/hadoop/BigData/flume-1.7.0-bin/bin/flume-ng agent -n a1 -c conf -f /home/hadoop/BigData/flume-1.7.0-bin/test_agent/agent_tailF2File.conf

 

Step3.3 测试 创建fileName.log并不断输入信息

$ echo "ABC" >> /home/hadoop/BigData/flume-1.7.0-bin/test/fileName.log
$ echo "CDF" >> /home/hadoop/BigData/flume-1.7.0-bin/test/fileName.log

此时,查看/home/hadoop/BigData/flume-1.7.0-bin/test/Sink01/即可发现相应的输出文件

 

 

 

分类上一篇:     分类下一篇:

Leave a Reply