首页 文章 精选 留言 我的

精选列表

搜索[部署]，共10000篇文章
优秀的个人博客,低调大师

Docker最佳实践-部署LNMP环境

标签（linux）： docker 笔者Q:972581034 交流群：605799367。有任何疑问可与笔者或加群交流 环境准备 [root@docker ~]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@docker ~]# free -h total used free shared buff/cache available Mem: 1.8G 209M 517M 8.9M 1.1G 1.3G Swap: 1.0G 0B 1.0G [root@docker ~]# hostname -I 10.0.0.250 172.16.1.250 systemctl disable firewalld.service systemctl stop firewalld.service iptables -F 安装yum-utils，它提供yum-config-manager实用程序 yum install -y yum-utils 添加稳定版本库 yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo 更新yum包索引 yum makecache fast 安装Docker社区版 yum install docker-ce -y 启动docker守护进程 systemctl enable docker systemctl start docker systemctl status docker 测试，，，网络延时问题可能会导致测试不成功，多测两次 docker run hello-world 查看镜像 c7是下面要用来启动容器的镜像，在centos镜像的基础上安装了常用软件包。 [root@docker ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE c7 latest 8af948261786 2 days ago 476 MB centos latest a8493f5f50ff 6 weeks ago 192 MB 这里需要一些相关的软件包和配置文件，这里给大家提供了链接：http://pan.baidu.com/s/1eR8vdXg 密码：l1sj 创建web01 docker run --network=host -itd --name=web01 -h web01 c7 docker attach web01 使用提供的软件包安装nginx #nginx yum localinstall -y nginx-1.10.2-1.x86_64.rpm nginx配置文件 cd /application/nginx/conf/ [root@web01 conf]# cat nginx.conf worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; include extra/blog.conf; } [root@web01 conf]# mkdir extra [root@web01 extra]# ls blog.conf 上传wordpress到站点根目录 [root@web01 extra]# cd /application/nginx/html/ [root@web01 html]# wget https://cn.wordpress.org/wordpress-4.7.4-zh_CN.tar.gz [root@web01 html]# tar xf wordpress-4.7.4-zh_CN.tar.gz [root@web01 html]# chown -R www.www /application/nginx/html 启动nginx,安装php并启动 /application/nginx/sbin/nginx chmod +x /etc/rc.local echo "/application/nginx/sbin/nginx" >> /etc/rc.local #php yum localinstall libiconv-devel-1.15-1.x86_64.rpm php-5.5.32-1.x86_64.rpm -y /application/php/sbin/php-fpm chmod +x /etc/rc.local echo "/application/php/sbin/php-fpm" >> /etc/rc.local 创建数据库容器 docker run --network=host -itd --name=db01 -h db01 c7 docker attach db01 安装并启动数据库 yum -y install mariadb mariadb-server mysql_install_db --user=mysql mysqld_safe &>/dev/null & /usr/bin/mysqladmin -u root password '123456' mysql -uroot -p123456 创建wordpress库和授权帐户 create database wordpress; grant all on wordpress.* to 'wordpress'@'localhost' identified by '123456'; grant all on wordpress.* to wordpress@'172.16.1.250' identified by '123456'; 登录浏览器测试 来个美女养养眼 docker常用命令http://www.cnblogs.com/Csir/p/6888474.html
2017-05-22
优秀的个人博客,低调大师

ELK环境部署（已踩坑）

已踩的坑： 由于版本的不一致，安装marvel时会导致kibana界面报红， 还有启动kibana失败，提示端口在使用，（og [17:26:44.467] [fatal] Error: listen EADDRINUSE IP:5601） 然后 ps -ef|grep kibana ps -ef|grep 5601 netstat -antp|grep 5601, 找到或者找到杀不掉，最后 尝试 使用 fuser -n tcp 5601 kill -9 端口 成功！！！ Logstash ， Elasticsearch Kibana ，的下载地址统一为https://www.elastic.co/downloads/ 本次使用的软件版本分别为： 1. elasticsearch2.4.4.tar.gz 2. kibana-4.6.4-linux-x86_64.tar.gz 3. logstash-2.0.0.tar.gz 4. ik1.10.4 一：Logstash安装 第一步.解压到指定目录， 使用tar xvzf *.tar.gz -C “path” 第二步：进入到解压后的根目录，使用bin/plugin install “插件名” 安装对应插件， 常用插件如下： logstash-filter-date logstash-filter-grok logstash-input-file logstash-input-stdin logstash-output-elasticsearch logstash-output-redis logstash-output-stdout 安装失败可以多重试，（看人品..） 第三步:配置 *.conf 文件启动时使用，注意配置中的input的文件必须拥有足够的权限，不然就等着失败… 二：Elasticsearch的安装 第一步：同样解压， 第二步：进入解压后的根目录下的config文件中，修改配置elasticsearch.xml文件 主要修改一下几点： cluster.name：必须保持集群内的各个节点此名称的一致性。 node.name: 每个节点起一个（类似 node-1,node-2…..) network.host:(填写本机的ip即可，这样就可以在windows也可以访问） discovery.zen.ping.unicast.hosts:（将里面的hosts换成对应的集群中所有节点的ip） 第三步：在$ES_HOME/下执行 bin/plugin install mobz/elasticsearch-head (有了它，就可以访问对应的web界面） 网址类似：http://”Ip“:9200/_plugin/head/ 第三步：安装Marvel 分别在elasticsearch和kibana下安装 Step 1: Install Marvel into Elasticsearch: bin/plugin install license bin/plugin install marvel-agent Step 2: Install Marvel into Kibana bin/kibana plugin –install elasticsearch/marvel/latest Step 3: Start Elasticsearch and Kibana bin/elasticsearch bin/kibana 第四步：Kibana的安装 1.同样解压后，进入config目录，配置 kibana.yml文件 ，其中只需要修改elastic.url属性即可。 改成本机的ip。 访问网址：http://”IP“:5601/ 第五步：Ik分词器的安装： https://github.com/medcl/elasticsearch-analysis-ik下载 我们的版本 1.10.4。 下载后的压缩包解压后进去是pom工程 分别执行如下命令： mvn clean mvn compile mvn package 2.当然这里是用maven对此工程进行编译，前提要安装maven Maven安装也很简单，下载包以后解压缩，配置环境变量即可 3.前面编译了插件以后会在target/releases目录下出现一个zip包 4.然后在es的根目录下的plugins里面建一个ik目录，将上面的zip文件拷进去，并解压 5.重启es完成。
2017-03-15
优秀的个人博客,低调大师

CentOS 6.8 实战部署JStorm集群

Alibaba JStorm 是一个强大的企业级流式计算引擎，是Apache Storm 的4倍性能， 可以自由切换行模式或mini-batch 模式，JStorm 不仅提供一个流式计算引擎， 还提供实时计算的完整解决方案， 涉及到更多的组件， 如jstorm-on-yarn, jstorm-on-docker, SQL Engine, Exactly-Once Framework 等等。 JStorm 是一个分布式实时计算引擎 JStorm 是一个类似Hadoop MapReduce的系统， 用户按照指定的接口实现一个任务，然后将这个任务递交给JStorm系统，JStorm将这个任务跑起来，并且按7 * 24小时运行起来，一旦中间一个Worker 发生意外故障， 调度器立即分配一个新的Worker替换这个失效的Worker。 因此，从应用的角度，JStorm应用是一种遵守某种编程规范的分布式应用。从系统角度， JStorm是一套类似MapReduce的调度系统。 从数据的角度，JStorm是一套基于流水线的消息处理机制。 实时计算现在是大数据领域中最火爆的一个方向，因为人们对数据的要求越来越高，实时性要求也越来越快，传统的Hadoop MapReduce，逐渐满足不了需求，因此在这个领域需求不断。 Storm组件和Hadoop组件对比 JStorm Hadoop 角色 Nimbus JobTracker Supervisor TaskTracker Worker Child 应用名称 Topology Job 编程接口 Spout/Bolt Mapper/Reducer 优点 在Storm和JStorm出现以前，市面上出现很多实时计算引擎，但自Storm和JStorm出现后，基本上可以说一统江湖： 究其优点: 开发非常迅速：接口简单，容易上手，只要遵守Topology、Spout和Bolt的编程规范即可开发出一个扩展性极好的应用，底层RPC、Worker之间冗余，数据分流之类的动作完全不用考虑 扩展性极好：当一级处理单元速度，直接配置一下并发数，即可线性扩展性能 健壮强：当Worker失效或机器出现故障时， 自动分配新的Worker替换失效Worker 数据准确性：可以采用Ack机制，保证数据不丢失。 如果对精度有更多一步要求，采用事务机制，保证数据准确。 实时性高： JStorm 的设计偏向单行记录，因此，在时延较同类产品更低 应用场景 JStorm处理数据的方式是基于消息的流水线处理， 因此特别适合无状态计算，也就是计算单元的依赖的数据全部在接受的消息中可以找到， 并且最好一个数据流不依赖另外一个数据流。 因此，常常用于: 日志分析，从日志中分析出特定的数据，并将分析的结果存入外部存储器如数据库。目前，主流日志分析技术就使用JStorm或Storm 管道系统， 将一个数据从一个系统传输到另外一个系统， 比如将数据库同步到Hadoop 消息转化器， 将接受到的消息按照某种格式进行转化，存储到另外一个系统如消息中间件 统计分析器， 从日志或消息中，提炼出某个字段，然后做count或sum计算，最后将统计值存入外部存储器。中间处理过程可能更复杂。 实时推荐系统， 将推荐算法运行在jstorm中，达到秒级的推荐效果 基本概念 首先，JStorm有点类似于Hadoop的MR（Map-Reduce），但是区别在于，hadoop的MR，提交到hadoop的MR job，执行完就结束了，进程就退出了，而一个JStorm任务（JStorm中称为topology），是7*24小时永远在运行的，除非用户主动kill。 JStorm组件 接下来是一张比较经典的Storm的大致的结构图（跟JStorm一样）： 图中的水龙头（好吧，有点俗）就被称作spout，闪电被称作bolt。 在JStorm的topology中，有两种组件：spout和bolt。 # spout spout代表输入的数据源，这个数据源可以是任意的，比如说kafaka，DB，HBase，甚至是HDFS等，JStorm从这个数据源中不断地读取数据，然后发送到下游的bolt中进行处理。 # bolt bolt代表处理逻辑，bolt收到消息之后，对消息做处理（即执行用户的业务逻辑），处理完以后，既可以将处理后的消息继续发送到下游的bolt，这样会形成一个处理流水线（pipeline，不过更精确的应该是个有向图）；也可以直接结束。 通常一个流水线的最后一个bolt，会做一些数据的存储工作，比如将实时计算出来的数据写入DB、HBase等，以供前台业务进行查询和展现。 组件的接口 JStorm框架对spout组件定义了一个接口：nextTuple，顾名思义，就是获取下一条消息。执行时，可以理解成JStorm框架会不停地调这个接口，以从数据源拉取数据并往bolt发送数据。 同时，bolt组件定义了一个接口：execute，这个接口就是用户用来处理业务逻辑的地方。 每一个topology，既可以有多个spout，代表同时从多个数据源接收消息，也可以多个bolt，来执行不同的业务逻辑。 调度和执行 接下来就是topology的调度和执行原理，对一个topology，JStorm最终会调度成一个或多个worker，每个worker即为一个真正的操作系统执行进程，分布到一个集群的一台或者多台机器上并行执行。 而每个worker中，又可以有多个task，分别代表一个执行线程。每个task就是上面提到的组件(component)的实现，要么是spout要么是bolt。 用户在提交一个topology的时候，会指定以下的一些执行参数： #总worker数 即总的进程数。举例来说，我提交一个topology，指定worker数为3，那么最后可能会有3个进程在执行。之所以是可能，是因为根据配置，JStorm有可能会添加内部的组件，如_acker或者__topology_master（这两个组件都是特殊的bolt），这样会导致最终执行的进程数大于用户指定的进程数。我们默认是如果用户设置的worker数小于10个，那么__topology_master 只是作为一个task存在，不独占worker；如果用户设置的worker数量大于等于10个，那么__topology_master作为一个task将独占一个worker #每个component的并行度 上面提到每个topology都可以包含多个spout和bolt，而每个spout和bolt都可以单独指定一个并行度(parallelism)，代表同时有多少个线程(task)来执行这个spout或bolt。 JStorm中，每一个执行线程都有一个task id，它从1开始递增，每一个component中的task id是连续的。 还是上面这个topology，它包含一个spout和一个bolt，spout的并行度为5，bolt并行度为10。那么我们最终会有15个线程来执行：5个spout执行线程，10个bolt执行线程。 这时spout的task id可能是1～5，bolt的task id可能是6～15，之所以是可能，是因为JStorm在调度的时候，并不保证task id一定是从spout开始，然后到bolt的。但是同一个component中的task id一定是连续的。 #每个component之间的关系 即用户需要去指定一个特定的spout发出的数据应该由哪些bolt来处理，或者说一个中间的bolt，它发出的数据应该被下游哪些bolt处理。 还是以上面的topology为例，它们会分布在3个进程中。JStorm使用了一种均匀的调度算法，因此在执行的时候，你会看到，每个进程分别都各有5个线程在执行。当然，由于spout是5个线程，不能均匀地分配到3个进程中，会出现一个进程只有1个spout线程的情况；同样地，也会出现一个进程中有4个bolt线程的情况。 在一个topology的运行过程中，如果一个进程（worker）挂掉了，JStorm检测到之后，会不断尝试重启这个进程，这就是7*24小时不间断执行的概念。 消息的通信 上面提到，spout的消息会发送给特定的bolt，bolt也可以发送给其他的bolt，那这之间是如何通信的呢？ 首先，从spout发送消息的时候，JStorm会计算出消息要发送的目标task id列表，然后看目标task id是在本进程中，还是其他进程中，如果是本进程中，那么就可以直接走进程内部通信（如直接将这个消息放入本进程中目标task的执行队列中）；如果是跨进程，那么JStorm会使用netty来将消息发送到目标task中。 实时计算结果输出 JStorm是7*24小时运行的，外部系统如果需要查询某个特定时间点的处理结果，并不会直接请求JStorm（当然，DRPC可以支持这种需求，但是性能并不是太好）。一般来说，在JStorm的spout或bolt中，都会有一个定时往外部存储写计算结果的逻辑，这样数据可以按照业务需求被实时或者近实时地存储起来，然后直接查询外部存储中的计算结果即可。 以上内容直接粘贴JStorm官网，切勿吐槽 二、 Jstorm 集群安装 1、系统环境准备 # OS: CentOS 6.8 mininal # host.ip: 10.1.1.78 aniutv-1 # host.ip: 10.1.1.80 aniutv-2 # host.ip: 10.1.1.97 aniutv-5 2、安装目录自定义 # jstorm : /opt/jstorm （源码安装）, zookeeper : /opt/zookeeper（源码安装） , java : /usr/java/jdk1.7.0_79 (rpm包安装) 3、zookeeper 集群安装 zookeeper 集群参考（http://blog.csdn.net/wh211212/article/details/56014983） 4、zeromq 安装 zeromq下载地址：http://zeromq.org/area:download/ 下载zeromq-4.2.1.tar.gz 到/usr/local/src cd /usr/local/src && tar -zxf zeromq-4.2.1.tar.gz -C /opt cd /opt/zeromq-4.2.1 && ./configure && make && sudo make install && sudo ldconfig 5、jzmq安装 cd /opt &&git clone https://github.com/nathanmarz/jzmq.git ./autogen.sh &&./configure &&make &&make install 6、JStorm安装 wget https://github.com/alibaba/jstorm/releases/download/2.1.1/jstorm-2.1.1.zip -P /usr/local/src cd /usr/local/src && unzip jstorm-2.1.1.zip -d /opt cd /opt && mv jstorm-2.1.1 jstorm # mkdir /opt/jstorm/jstorm_data echo '# jstorm env' >> ~/.bashrc echo 'export JSTORM_HOME=/opt/jstorm' >> ~/.bashrc echo 'export PATH=$PATH:$JSTORM_HOME/bin' >> ~/.bashrc source ~/.bashrc # JStorm 配置 sed -i /'storm.zookeeper.servers:/a\ - "10.1.1.78"' /opt/jstorm/conf/storm.yaml sed -i /'storm.zookeeper.servers:/a\ - "10.1.1.80"' /opt/jstorm/conf/storm.yaml sed -i /'storm.zookeeper.servers:/a\ - "10.1.1.97"' /opt/jstorm/conf/storm.yaml sed -i /'storm.zookeeper.root/a\ nimbus.host: "10.1.1.78"' /opt/jstorm/conf/storm.yaml<> 配置项： storm.zookeeper.servers: 表示zookeeper 的地址， nimbus.host: 表示nimbus的地址 storm.zookeeper.root: 表示JStorm在zookeeper中的根目录，当多个JStorm共享一个zookeeper时，需要设置该选项，默认即为“/jstorm” storm.local.dir: 表示JStorm临时数据存放目录，需要保证JStorm程序对该目录有写权限 java.library.path: Zeromq 和java zeromq library的安装目录，默认"/usr/local/lib:/opt/local/lib:/usr/lib" supervisor.slots.ports: 表示Supervisor 提供的端口Slot列表，注意不要和其他端口发生冲突，默认是68xx，而Storm的是67xx topology.enable.classloader: false, 默认关闭classloader，如果应用的jar与JStorm的依赖的jar发生冲突，比如应用使用thrift9，但jstorm使用thrift7时，就需要打开classloader。建议在集群级别上默认关闭，在具体需要隔离的topology上打开这个选项。 # 下面命令只需要在安装 jstorm_ui 和提交jar节点的机器上面执行即可 mkdir ~/.jstorm cp -f $JSTORM_HOME/conf/storm.yaml ~/.jstorm 7、安装JStorm Web UI 强制使用tomcat7.0或以上版本，切记拷贝~/.jstorm/storm.yaml，Web UI 可以和Nimbus在同一个节点上 mkdir ~/.jstorm cp -f $JSTORM_HOME/conf/storm.yaml ~/.jstorm 下载tomcat 7.x （以apache-tomcat-7.0.37 为例） tar -xzf apache-tomcat-7.0.75.tar.gz cd apache-tomcat-7.0.75 cd webapps cp $JSTORM_HOME/jstorm-ui-2.1.1.war ./ mv ROOT ROOT.old ln -s jstorm-ui-2.1.1 ROOT # 另外不是 ln -s jstorm-ui-2.1.1.war ROOT 这个要小心 cd ../bin ./startup.sh 8、JStorm启动 在nimbus 节点（10.1.1.78）上执行 “nohup jstorm nimbus &”, 查看$JSTORM_HOME/logs/nimbus.log检查有无错误 在supervisor节点(10.1.1.78,10.1.1.80,10.1.1.97)上执行 “nohup jstorm supervisor &”, 查看$JSTORM_HOME/logs/supervisor.log检查有无错误 9、JStorm Web UI JStorm集群启动成功截图如下： # JStorm 集群安装问题总结 1、注意/etc/hosts设置，添加相对应的ip hostname 2、设置ssh免密操作（此步骤在zookeeper集群完成） 3、注意各服务的环境变量设置
2017-02-24
优秀的个人博客,低调大师

Ansible批量部署zabbix-agent

Ansible的目录树结构如下： # tree zabbix/ zabbix/ ├──defaults ├──files │└──zabbix.tar.gz ├──handlers ├──meta ├──tasks │└──main.yml ├──templates │└──zabbix_agentd.conf.j2 └──vars └──main.yaml zabbix.tar.gz:tar包是编译安装完zabbix后打的tar包 zabbix_agentd.conf.j2:编译安装完zabbix，copy的zabbix_agentd.conf文件，略微添加点东西,后面会展示。 zabbix_agentd：文件也是编译安装完zabbix，copy的启动脚本文件，修改了里面的zabbix_home变量。 zabbix playbook文件如下： catzabbix.yml -hosts:all roles: -zabbix tasks文件内容如下： #catmain.yml -name:scpzabbix到client copy:src=zabbix.tar.gzdest={{zabbix_home}} -name:scpzabbix_aegntd到client copy:src=/tmp/zabbix_agentddest=/etc/init.d/mode=777 -name:tarzxfzabbix.tar.gz解压 shell:cd{{zabbix_home}}&&tar-zxfzabbix.tar.gz -name:Copyzabbix_agent.confConfigfile template:> src=zabbix_agentd.conf.j2 dest=/usr/local/zabbix/etc/zabbix_agentd.conf mode=0644owner=rootgroup=root -name:restartzabbix service:name=zabbix_agentdstate=restarted vars的内容如下： #catmain.yaml zabbix_home:/usr/local zabbix_agentd.conf.j2模板的文件内容如下： #grep-v'^#'zabbix_agentd.conf.j2|grep-v'^$' LogFile=/tmp/zabbix_agentd.log EnableRemoteCommands=1 Server=10.74.246.70 ListenPort=10050 ServerActive=10.74.246.70 Hostname={{ansible_default_ipv4.address}} AllowRoot=1 Include=/usr/local/zabbix/etc/zabbix_agentd.conf.d/*.conf UnsafeUserParameters=1 UserParameter=tomcat_port,/usr/local/zabbix/share/zabbix/alertscripts/tomcat_port.sh$1 UserParameter=tcp[*],/usr/local/zabbix/share/zabbix/alertscripts/tcp_connection.sh$1 UserParameter=nginx_port,/usr/local/zabbix/share/zabbix/alertscripts/nginx_port.sh$1 UserParameter=erro_log[*],/usr/local/zabbix/share/zabbix/alertscripts/error_log.sh$1 然后就可以定义hosts文件，批量操作安装zabbix_agent。 对于某些特定的主机，需要批量推送特定的zabbix_agentd.conf文件，可以修改完zabbix_agentd.conf.j2模板，然后使用ansible推送： #ansible-playbookzabbix.yml--start-at-task='Copyzabbix_agent.confConfigfile' 当然也可以指定推送那些机器，后面-l指定就行 #ansible-playbookzabbix.yml--start-at-task='Copyzabbix_agent.confConfigfile'-l192.168.121.128
2017-02-14
优秀的个人博客,低调大师

视频 - 在 VirtualBox 中部署 OpenStack

大家新年好，CloudMan 今天给大家带来一件新年礼物。 一直以来大家都反馈 OpenStack 学习有两大障碍：1. 实验环境难搭2. 体系复杂，难道大今天我就先帮大家解决环境问题。前两天我抽空在笔记本 VirtualBox 中搭建了两节点 N 版 OpenStack 环境，并全程录制了视频。经过几天的剪辑和加工，终于赶在今天出炉，希望对大家有帮助。 为方便大家照着操作，我做了四个视频，放在了http://cloudman.cc上。 （由于 cloudman.cc 目前托管在 github上，网页第一次加载稍微会有些慢，望见谅。不过视频还是很快的）
2017-01-02
优秀的个人博客,低调大师

[Flume]安装，部署与应用案例

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/SunnyYoona/article/details/52585478 1. 官网 http://flume.apache.org/ 2. 下载 http://flume.apache.org/download.html 3. 安装 3.1 将下载的flume包，解压到/opt目录中 3.2 创建flume-env.sh 配置文件 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume-env.sh.template flume-env.sh 3.3 修改 flume-env.sh 配置文件，主要是JAVA_HOME变量设置 # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced # during Flume startup. # Enviroment variables can be set here. # export JAVA_HOME=/usr/lib/jvm/java-6-sun export JAVA_HOME=/opt/jdk1.8.0_91 # Give Flume more memory and pre-allocate, enable remote monitoring via JMX # export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote" # Note that the Flume conf directory is always included in the classpath. #FLUME_CLASSPATH="" 3.4 验证是否安装成功 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng version Flume 1.6.0 Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080 Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015 From source with checksum b29e416802ce9ece3269d34233baf43f 出现上面信息，表示安装成功了。 4. 案例 4.1 案例一 Avro Avro可以发送一个给定的文件给Flume，Avro 源使用AVRO RPC机制。 4.1.1 创建agent 配置文件 根据模板文件创建配置文件： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume-conf.properties.template flume.conf 4.1.2 配置agent配置文件 当你运行一个agent的时候，需要通过-f 选项来告诉Flume使用哪个配置文件。让我们看一个基本的例子，复制下面代码并粘贴到conf/flume.conf文件中。 # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = avro agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 4141 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.1.3 启动flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume.conf -n agent1 -Dflume.root.logger=INFO,console 4.1.4 创建指定文件 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ sudo touch log.00 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ sudo vim log.00 4.1.5 使用avro-client发送文件 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng avro-client -c . -H 0.0.0.0 -p 4141 -F ../log.00 4.1.6 查看信息 在启动agent的控制窗口，可以看到一下信息，注意最后一行： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 10:29:27 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 10:29:27 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume.conf 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 10:29:27 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 10:29:27 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 10:29:27 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 10:29:27 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type avro 16/09/19 10:29:27 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 10:29:27 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 10:29:27 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:Avro source avro-source1: { bindAddress: 0.0.0.0, port: 4141 } }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@453e9d5e counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 10:29:27 INFO node.Application: Starting Channel ch1 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 10:29:27 INFO node.Application: Starting Sink logger-sink1 16/09/19 10:29:27 INFO node.Application: Starting Source avro-source1 16/09/19 10:29:27 INFO source.AvroSource: Starting Avro source avro-source1: { bindAddress: 0.0.0.0, port: 4141 }... 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 10:29:27 INFO source.AvroSource: Avro source avro-source1 started. 16/09/19 10:36:32 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 => /127.0.0.1:4141] OPEN 16/09/19 10:36:32 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 => /127.0.0.1:4141] BOUND: /127.0.0.1:4141 16/09/19 10:36:32 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 => /127.0.0.1:4141] CONNECTED: /127.0.0.1:42708 16/09/19 10:36:33 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 :> /127.0.0.1:4141] DISCONNECTED 16/09/19 10:36:33 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 :> /127.0.0.1:4141] UNBOUND 16/09/19 10:36:33 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 :> /127.0.0.1:4141] CLOSED 16/09/19 10:36:33 INFO ipc.NettyServer: Connection to /127.0.0.1:42708 disconnected. 16/09/19 10:36:37 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 Hello Flume } 4.2 案例二 Spool Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点： （1）拷贝到spool目录下的文件不可以再打开编辑。 （2）spool目录下不可包含相应的子目录 4.2.1 创建配置文件flume-spool.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-spool.conf 进行一下配置： # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = spooldir agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.spoolDir = /home/xiaosi/logs/ agent1.sources.avro-source1.fileHeader = true agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 4141 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 对/home/xiaosi/logs目录进行监控。 4.2.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-spool.conf -n agent1 -Dflume.root.logger=INFO,console 4.2.3 追加文件到监控目录 xiaosi@Qunar:~$ echo "Hello Flume first" > /home/xiaosi/logs/flume-log-1.log xiaosi@Qunar:~$ echo "Hello Flume second" > /home/xiaosi/logs/flume-log-2.log 4.2.4 查看信息 在启动agent的控制窗口，可以看到一下信息，注意最后两行： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-spool.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Including Hive libraries found via (/opt/apache-hive-2.0.0-bin) for Hive access ... org.apache.flume.node.Application -f ../conf/flume-spool.conf -n agent1 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 11:29:52 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 11:29:52 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-spool.conf 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 11:29:52 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 11:29:52 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 11:29:52 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 11:29:52 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type spooldir 16/09/19 11:29:52 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 11:29:52 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 11:29:52 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:Spool Directory source avro-source1: { spoolDir: /home/xiaosi/logs/ } }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4f5f731e counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 11:29:52 INFO node.Application: Starting Channel ch1 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 11:29:52 INFO node.Application: Starting Sink logger-sink1 16/09/19 11:29:52 INFO node.Application: Starting Source avro-source1 16/09/19 11:29:52 INFO source.SpoolDirectorySource: SpoolDirectorySource source starting with directory: /home/xiaosi/logs/ 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 11:30:06 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one. 16/09/19 11:30:06 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /home/xiaosi/logs/flume-log-1.log to /home/xiaosi/logs/flume-log-1.log.COMPLETED 16/09/19 11:30:07 INFO sink.LoggerSink: Event: { headers:{file=/home/xiaosi/logs/flume-log-1.log} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 66 69 72 73 Hello Flume firs } 16/09/19 11:30:21 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one. 16/09/19 11:30:21 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /home/xiaosi/logs/flume-log-2.log to /home/xiaosi/logs/flume-log-2.log.COMPLETED 16/09/19 11:30:22 INFO sink.LoggerSink: Event: { headers:{file=/home/xiaosi/logs/flume-log-2.log} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 73 65 63 6F Hello Flume seco } 4.3 案例三 Exec EXEC执行一个给定的命令获得输出的源，如果要使用tail命令，必选使得file足够大才能看到输出内容 4.3.1 创建配置文件flume-exec.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-exec.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = exec agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.command = tail -F /home/xiaosi/logs/flume-log-exec.log agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 4141 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.2.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-exec.conf -n agent1 -Dflume.root.logger=INFO,console 4.2.3 执行tail命令 向文件中进行追加数据，生成足够多的数据： #! /bin/sh for index in {1..100} do echo "Hello Flume $index" >> /home/xiaosi/logs/flume-log-exec.log done 同时对文件使用tail 命令操作： xiaosi@Qunar:~$ tail -F /home/xiaosi/logs/flume-log-exec.log Hello Flume 1 Hello Flume 2 Hello Flume 3 Hello Flume 4 ... 4.3.4 查看信息 在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-exec.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 12:01:28 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 12:01:28 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-exec.conf 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 12:01:28 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 12:01:28 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 12:01:28 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 12:01:28 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type exec 16/09/19 12:01:28 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 12:01:28 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 12:01:28 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@242d6c8b counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 12:01:28 INFO node.Application: Starting Channel ch1 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 12:01:28 INFO node.Application: Starting Sink logger-sink1 16/09/19 12:01:28 INFO node.Application: Starting Source avro-source1 16/09/19 12:01:28 INFO source.ExecSource: Exec source starting with command:tail -F /home/xiaosi/logs/flume-log-exec.log 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 31 Hello Flume 1 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 32 Hello Flume 2 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 33 Hello Flume 3 } ... 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 39 38 Hello Flume 98 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 39 39 Hello Flume 99 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 31 30 30 Hello Flume 100 } 4.4 案例四 Syslogtcp Syslogtcp监听TCP的端口做为数据源 4.4.1 创建配置文件flume-tcp.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-tcp.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = syslogtcp agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.host = localhost #agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 5140 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.4.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-tcp.conf -n agent1 -Dflume.root.logger=INFO,console 4.4.3 测试产生syslog xiaosi@Qunar:~$ echo "hello flume tcp" | nc localhost 5140 4.4.4 查看信息 在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-tcp.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 12:10:15 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 12:10:15 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-tcp.conf 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 12:10:15 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 12:10:15 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 12:10:15 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 12:10:15 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type syslogtcp 16/09/19 12:10:15 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 12:10:15 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 12:10:15 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogTcpSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@38aab021 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 12:10:15 INFO node.Application: Starting Channel ch1 16/09/19 12:10:16 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 12:10:16 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 12:10:16 INFO node.Application: Starting Sink logger-sink1 16/09/19 12:10:16 INFO node.Application: Starting Source avro-source1 16/09/19 12:10:16 INFO source.SyslogTcpSource: Syslog TCP Source starting... 16/09/19 12:10:50 WARN source.SyslogUtils: Event created from Invalid Syslog data. 16/09/19 12:10:54 INFO sink.LoggerSink: Event: { headers:{Severity=0, Facility=0, flume.syslog.status=Invalid} body: 68 65 6C 6C 6F 20 66 6C 75 6D 65 20 74 63 70 hello flume tcp } 4.5 案例五 JSONHandler 4.5.1 创建配置文件flume-json.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-json.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = org.apache.flume.source.http.HTTPSource agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.port = 8888 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.5.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-json.conf -n agent1 -Dflume.root.logger=INFO,console 4.5.3 生成JSON 格式的POST request xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ curl -X POST -d '[{ "headers" :{"a":"a1", "b":"b1"}, "body":"flume_json_boy"}]' http://localhost:8888 4.5.4 查看信息 在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-json.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 13:21:28 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 13:21:28 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-json.conf 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 13:21:28 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 13:21:28 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 13:21:28 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 13:21:28 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type org.apache.flume.source.http.HTTPSource 16/09/19 13:21:28 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 13:21:28 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 13:21:28 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@136bcdd0 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 13:21:28 INFO node.Application: Starting Channel ch1 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 13:21:28 INFO node.Application: Starting Sink logger-sink1 16/09/19 13:21:28 INFO node.Application: Starting Source avro-source1 16/09/19 13:21:28 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 16/09/19 13:21:28 INFO mortbay.log: jetty-6.1.26 16/09/19 13:21:28 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:8888 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 13:21:32 INFO sink.LoggerSink: Event: { headers:{a=a1, b=b1} body: 66 6C 75 6D 65 5F 6A 73 6F 6E 5F 62 6F 79 flume_json_boy } 4.6 案例六 Hadoop Sink Syslogtcp监听TCP的端口做为数据源，并将监听的数据存储在HDFS中 4.6.1 创建配置文件flume-hadoop.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-hadoop.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = syslogtcp agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.host = localhost agent1.sources.avro-source1.port = 5140 # sink agent1.sinks.logger-sink1.type = hdfs agent1.sinks.logger-sink1.channel = ch1 agent1.sinks.logger-sink1.hdfs.path = hdfs://localhost:9000/user/xiaosi/data agent1.sinks.logger-sink1.hdfs.filePrefix = SysLog agent1.sinks.logger-sink1.hdfs.round = true agent1.sinks.logger-sink1.hdfs.roundValue = 10 agent1.sinks.logger-sink1.hdfs.roundUnit = minute # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.6.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-hadoop.conf -n agent1 -Dflume.root.logger=INFO,console 4.6.3 测试产生syslog xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ echo "Hello Flume -> Hadoop one" | nc localhost 5140 4.6.4 查看信息 在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-hadoop.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 13:34:58 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 13:34:58 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-hadoop.conf 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 13:34:58 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 13:34:58 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 13:34:58 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 13:34:58 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type syslogtcp 16/09/19 13:34:58 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: hdfs 16/09/19 13:34:58 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 13:34:58 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogTcpSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@569671b3 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 13:34:58 INFO node.Application: Starting Channel ch1 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 13:34:58 INFO node.Application: Starting Sink logger-sink1 16/09/19 13:34:58 INFO node.Application: Starting Source avro-source1 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: logger-sink1: Successfully registered new MBean. 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: logger-sink1 started 16/09/19 13:34:58 INFO source.SyslogTcpSource: Syslog TCP Source starting... 16/09/19 13:35:06 WARN source.SyslogUtils: Event created from Invalid Syslog data. 16/09/19 13:35:07 INFO hdfs.HDFSSequenceFile: writeFormat = Writable, UseRawLocalFileSystem = false 16/09/19 13:35:07 INFO hdfs.BucketWriter: Creating hdfs://localhost:9000/user/xiaosi/data/SysLog.1474263307767.tmp 4.6.5 查看HDFS xiaosi@Qunar:/opt/hadoop-2.7.2/sbin$ hadoop fs -ls /user/xiaosi/data Found 3 items -rw-r--r-- 1 xiaosi supergroup 141 2016-09-19 13:35 /user/xiaosi/data/SysLog.1474263307767 -rw-r--r-- 1 xiaosi supergroup 1350 2016-07-28 14:10 /user/xiaosi/data/mysql-result.txt -rw-r--r-- 3 xiaosi supergroup 26 2016-07-30 22:47 /user/xiaosi/data/num.txt xiaosi@Qunar:/opt/hadoop-2.7.2/sbin$ hadoop fs -text /user/xiaosi/data/SysLog.1474263307767 1474263309104 48 65 6c 6c 6f 20 46 6c 75 6d 65 20 2d 3e 20 48 61 64 6f 6f 70 20 20 6f 6e 65 xiaosi@Qunar:/opt/hadoop-2.7.2/sbin$ hadoop fs -cat /user/xiaosi/data/SysLog.1474263307767 SEQ
2016-09-19

资源下载

更多资源
优质分享App

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

时间: 2025-12-03 大小: 4.36MB 下载: 23次
Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

时间: 2025-12-21 大小: 189MB 下载: 2次
Spring

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

时间: 2025-12-17 大小: 5MB 下载: 1次
Rocky Linux

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

时间: 2026-03-02 大小: 1.42GB 下载: 3次
热门标签
Jdk25 Docker SpringCloud Kubernetes OpenAi Nacos3 DeepSeek Rocky HarmonyOS6 Redis SpringBoot4 Service Mesh Gemini3

欢迎您来访！

登录后，您将获得更多功能！
热门文章
联系我们

有任何问题或合作意向欢迎联系我们

Email: 99873273@qq.com

QQ: 99873273

用户登录
用户注册