04【在线日志分析】之Flume Agent的3台收集+1台聚合到hdfs的搭建-低调大师

04【在线日志分析】之Flume Agent的3台收集+1台聚合到hdfs的搭建

2017-01-06 668

【log收集】:
机器名称        服务名称    用户
flume-agent-01: namenode    hdfs
flume-agent-02: datanode    hdfs
flume-agent-03: datanode    hdfs

【log聚合】:
机器名称用户
sht-sgmhadoopcm-01(172.16.101.54) root

【sink到hdfs】:
hdfs://172.16.101.56:8020/testwjp/

1.下载apache-flume-1.7.0-bin.tar.gz
[hdfs@flume-agent-01 tmp]$ wget http://www-eu.apache.org/dist/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
--2017-01-04 20:40:10-- http://www-eu.apache.org/dist/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
Resolving www-eu.apache.org... 88.198.26.2, 2a01:4f8:130:2192::2
Connecting to www-eu.apache.org|88.198.26.2|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 55711670 (53M) [application/x-gzip]
Saving to: “apache-flume-1.7.0-bin.tar.gz”

100%[===============================================================================================================================================================================================>] 55,711,670 473K/s in 74s

2017-01-04 20:41:25 (733 KB/s) - “apache-flume-1.7.0-bin.tar.gz” saved [55711670/55711670]

2.解压重命名
[hdfs@flume-agent-01 tmp]$
[hdfs@flume-agent-01 tmp]$ tar -xzvf apache-flume-1.7.0-bin.tar.gz
[hdfs@flume-agent-01 tmp]$ mv apache-flume-1.7.0-bin flume-ng
[hdfs@flume-agent-01 tmp]$ cd flume-ng/conf

3.拷贝flume环境配置和agent配置文件
[hdfs@flume-agent-01 tmp]$ cp flume-env.sh.template flume-env.sh
[hdfs@flume-agent-01 tmp]$ cp flume-conf.properties.template exec_memory_avro.properties

4.添加hdfs用户的环境变量文件
[hdfs@flume-agent-01 tmp]$ cd
[hdfs@flume-agent-01 ~]$ ls -la
total 24
drwxr-xr-x   3 hdfs hadoop 4096 Jul 8 14:05 .
drwxr-xr-x. 35 root root   4096 Dec 10 2015 ..
-rw-------   1 hdfs hdfs   4471 Jul 8 17:22 .bash_history
drwxrwxrwt   2 hdfs hadoop 4096 Nov 19 2014 cache
-rw-------   1 hdfs hdfs   3131 Jul 8 14:05 .viminfo
[hdfs@flume-agent-01 ~]$ cp /etc/skel/.* ./
cp: omitting directory `/etc/skel/.'
cp: omitting directory `/etc/skel/..'
[hdfs@flume-agent-01 ~]$ ls -la
total 36
drwxr-xr-x   3 hdfs hadoop 4096 Jan 4 20:49 .
drwxr-xr-x. 35 root root   4096 Dec 10 2015 ..
-rw-------   1 hdfs hdfs   4471 Jul 8 17:22 .bash_history
-rw-r--r--   1 hdfs hdfs     18 Jan 4 20:49 .bash_logout
-rw-r--r--   1 hdfs hdfs    176 Jan 4 20:49 .bash_profile
-rw-r--r--   1 hdfs hdfs    124 Jan 4 20:49 .bashrc
drwxrwxrwt   2 hdfs hadoop 4096 Nov 19 2014 cache
-rw-------   1 hdfs hdfs   3131 Jul 8 14:05 .viminfo

5.添加flume的环境变量
[hdfs@flume-agent-01 ~]$ vi .bash_profile

export FLUME_HOME=/tmp/flume-ng
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$PATH:$FLUME_HOME/bin
[hdfs@flume-agent-01 ~]$ . .bash_profile

6.修改flume环境配置文件
[hdfs@flume-agent-01 conf]$ vi flume-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_25

7.将基于Flume-ng Exec Source开发自定义插件AdvancedExecSource的AdvancedExecSource.jar包上传到$FLUME_HOME/lib/
http://blog.itpub.net/30089851/viewspace-2131995/

[hdfs@LogshedNameNodeLogcollector lib]$ pwd
/tmp/flume-ng/lib
[hdfs@LogshedNameNodeLogcollector lib]$ ll AdvancedExecSource.jar
-rw-r--r-- 1 hdfs hdfs 10618 Jan 5 23:50 AdvancedExecSource.jar
[hdfs@LogshedNameNodeLogcollector lib]$

8.修改flume的agent配置文件
[hdfs@flume-agent-01 conf]$ vi exec_memory_avro.properties
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the custom exec source
a1.sources.r1.type = com.onlinelog.analysis.AdvancedExecSource
a1.sources.r1.command = tail -f /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-NAMENODE-flume-agent-01.log.out
a1.sources.r1.hostname = flume-agent-01
a1.sources.r1.servicename = namenode

# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = 172.16.101.54
a1.sinks.k1.port = 4545

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.keep-alive = 60
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 2000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

9.将flume-agent-01的flume-ng打包,scp到flume-agent-02/03 和 sht-sgmhadoopcm-01(172.16.101.54)
[hdfs@flume-agent-01 tmp]$ zip -r flume-ng.zip flume-ng/*

[jpwu@flume-agent-01 ~]$ scp /tmp/flume-ng.zip flume-agent-02:/tmp/
[jpwu@flume-agent-01 ~]$ scp /tmp/flume-ng.zip flume-agent-03:/tmp/
[jpwu@flume-agent-01 ~]$ scp /tmp/flume-ng.zip sht-sgmhadoopcm-01:/tmp/

10.在flume-agent-02配置hdfs用户环境变量和解压，修改agent配置文件
[hdfs@flume-agent-02 ~]$ cp /etc/skel/.* ./
cp: omitting directory `/etc/skel/.'
cp: omitting directory `/etc/skel/..'
[hdfs@flume-agent-02 ~]$ vi .bash_profile
export FLUME_HOME=/tmp/flume-ng
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$PATH:$FLUME_HOME/bin
[hdfs@flume-agent-02 ~]$ . .bash_profile

[hdfs@flume-agent-02 tmp]$ unzip flume-ng.zip
[hdfs@flume-agent-02 tmp]$ cd flume-ng/conf

##修改以下参数即可
[hdfs@flume-agent-02 conf]$ vi exec_memory_avro.properties
a1.sources.r1.command = tail -f /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-DATANODE-flume-agent-02.log.out
a1.sources.r1.hostname = flume-agent-02
a1.sources.r1.servicename = datanode

###要检查flume-env.sh的JAVA_HOME目录是否存在

11.在flume-agent-03配置hdfs用户环境变量和解压，修改agent配置文件
[hdfs@flume-agent-03 ~]$ cp /etc/skel/.* ./
cp: omitting directory `/etc/skel/.'
cp: omitting directory `/etc/skel/..'
[hdfs@flume-agent-03 ~]$ vi .bash_profile
export FLUME_HOME=/tmp/flume-ng
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$PATH:$FLUME_HOME/bin
[hdfs@flume-agent-03 ~]$ . .bash_profile

[hdfs@flume-agent-03 tmp]$ unzip flume-ng.zip
[hdfs@flume-agent-03 tmp]$ cd flume-ng/conf

##修改以下参数即可
[hdfs@flume-agent-03 conf]$ vi exec_memory_avro.properties
a1.sources.r1.command = tail -f /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-DATANODE-flume-agent-03.log.out
a1.sources.r1.hostname = flume-agent-03
a1.sources.r1.servicename = datanode

###要检查flume-env.sh的JAVA_HOME目录是否存在

12.聚合端 sht-sgmhadoopcm-01，配置root用户环境变量和解压，修改agent配置文件
[root@sht-sgmhadoopcm-01 tmp]# vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export FLUME_HOME=/tmp/flume-ng
export FLUME_CONF_DIR=$FLUME_HOME/conf

export PATH=$FLUME_HOME/bin:$JAVA_HOME/bin:$PATH
[root@sht-sgmhadoopcm-01 tmp]# source /etc/profile
[root@sht-sgmhadoopcm-01 tmp]#

[root@sht-sgmhadoopcm-01 tmp]# unzip flume-ng.zip
[root@sht-sgmhadoopcm-01 tmp]# cd flume-ng/conf

[root@sht-sgmhadoopcm-01 conf]# vi flume-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

###测试: 先聚合, sink到hdfs端
[root@sht-sgmhadoopcm-01 conf]# vi avro_memory_hdfs.properties
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 172.16.101.54
a1.sources.r1.port = 4545

# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://172.16.101.56:8020/testwjp/
a1.sinks.k1.hdfs.filePrefix = logs
a1.sinks.k1.hdfs.inUsePrefix = .

a1.sinks.k1.hdfs.rollInterval = 0
### roll 16 m = 16777216 bytes
a1.sinks.k1.hdfs.rollSize = 1048576
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.batchSize = 6000

a1.sinks.k1.hdfs.writeFormat = text
a1.sinks.k1.hdfs.fileType = DataStream

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.keep-alive = 90
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 6000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

13.后台启动
[root@sht-sgmhadoopcm-01 flume-ng]# source /etc/profile
[hdfs@flume-agent-01 flume-ng]$ . ~/.bash_profile
[hdfs@flume-agent-02 flume-ng]$ . ~/.bash_profile
[hdfs@flume-agent-03 flume-ng]$ . ~/.bash_profile

[root@sht-sgmhadoopnn-01 flume-ng]# nohup flume-ng agent -c conf -f /tmp/flume-ng/conf/avro_memory_hdfs.properties -n a1 -Dflume.root.logger=INFO,console &

[hdfs@flume-agent-01 flume-ng]$ nohup flume-ng agent -c /tmp/flume-ng/conf -f /tmp/flume-ng/conf/exec_memory_avro.properties -n a1 -Dflume.root.logger=INFO,console &
[hdfs@flume-agent-01 flume-ng]$ nohup flume-ng agent -c /tmp/flume-ng/conf -f /tmp/flume-ng/conf/exec_memory_avro.properties -n a1 -Dflume.root.logger=INFO,console &
[hdfs@flume-agent-01 flume-ng]$ nohup flume-ng agent -c /tmp/flume-ng/conf -f /tmp/flume-ng/conf/exec_memory_avro.properties -n a1 -Dflume.root.logger=INFO,console &

14.校验：将集群的日志下载到本地，打开查看即可(略)

------------------------------------------------------------------------------------------------------------------------------------------------

【备注】:
1.错误1 flume-ng安装的机器上没有hadoop环境，所以假如sink到hdfs话，需要用到hdfs的jar包
[ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:146)] Failed to start agent
because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType

只需在其他安装hadoop机器上搜索以下5个jar包，拷贝到$FLUME_HOME/lib目录即可。
搜索方法: find $HADOOP_HOME/ -name commons-configuration*.jar

commons-configuration-1.6.jar
hadoop-auth-2.7.3.jar
hadoop-common-2.7.3.jar
hadoop-hdfs-2.7.3.jar
hadoop-mapreduce-client-core-2.7.3.jar
protobuf-java-2.5.0.jar
htrace-core-3.1.0-incubating.jar
commons-io-2.4.jar

2.错误2 无法加载自定义插件的类 Unable to load source type: com.onlinelog.analysis.AdvancedExecSource
2017-01-06 21:10:48,278 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)] Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.onlinelog.analysis.AdvancedExecSource, class: com.onlinelog.analysis.AdvancedExecSource

执行hdfs或者root用户的环境变量即可
[root@sht-sgmhadoopcm-01 flume-ng]# source /etc/profile
[hdfs@flume-agent-01 flume-ng]$ . ~/.bash_profile
[hdfs@flume-agent-02 flume-ng]$ . ~/.bash_profile
[hdfs@flume-agent-03 flume-ng]$ . ~/.bash_profile

微信关注我们

原文链接：https://yq.aliyun.com/articles/578754

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

00【在线日志分析】之项目概述和GitHub项目地址

1.前期基本架构图2.最终基本架构图 3.版本 Flume: 1.7 Hadoop: 2.7.3Scala: 2.11Kafka: 0.10.1.0 Spark: 2.0.2 InfluxDB: 1.2.0 Grafana: 4.1.1 maven: 3.3.9 4.主要目的主要是想基于Exec Source开发自定义插件AdvancedExecSource，将机器名称和服务名称添加到cdh 服务的角色log数据的每一行前面，则格式为：机器名称服务名称年月日时分秒.毫秒日志级别日志信息；然后在后面的spark streaming 实时计算我们所需求：比如统计每台机器的服务的每秒出现的error次数、统计每5秒的warn，error次数等等；来实时可视化展示和邮件短信、微信企业号通知。其实主要我们现在的很多监控服务基本达不到秒级的通知，都为5分钟等等，为了方便我们自己的维护；其实对一些即将出现的问题可以提前预知；其实最主要可以有效扩展到实时计算数据库级别日志，比如MySQL慢查询日志，nginx，tomcat，linux的系统级别日志等等。 5.大概流程1...

2017-01-07

619

Partitioners负责划分Maper输出的中间键值对的key，分配中间键值对到不同的Reducer。Maper输出的中间结果交给指定的Partitioner，确保中间结果分发到指定的Reduce任务。在每个Reducer中，键按排序顺序处理（Within each reducer, keys are processed in sorted order）。Combiners是MapReduce中的一个优化，允许在shuffle和排序阶段之前在本地进行聚合。Combiners的首要目标是通过最小化键值对的数量来节省尽可能多的带宽，键值对将通过网络在mappers和reducers之间进行shuffle操作（The primary goal of combiners is to save as much bandwidth as possible by minimizing the number of key/value pairs that will be shuffled across the network between mappers and reducers）。我们可以把C...

2017-01-07

580

资源下载

更多资源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。