搜索[微服务与微计算]结果-低调大师优秀个人博客

TensorFlow中文社区---下载与安装

转自：http://www.tensorfly.cn/tfdoc/get_started/os_setup.html 下载与安装你可以使用我们提供的二进制包, 或者使用源代码, 安装 TensorFlow

2017-03-24

Android RxAndroid与RxJava之intervalRange

intervalRange以一个例子说明：从1开始输出10个数据，延迟0秒执行，每隔2秒执行一次： package zhangphil.app; import android.app.Activity; import android.os.Bundle; import android.support.annotation.Nullable; import android.util.Log; import java.util.UUID; import java.util.concurrent.TimeUnit; import io.reactivex.Observable; import io.reactivex.functions.Consumer; public class MainActivity extends Activity { private final String TAG = String.valueOf(UUID.randomUUID()); @Override public void onCreate(@Nullable Bundle savedInstanceState) { super.onCreate(savedInstanceState); Observable.intervalRange(1, 10, 0, 2, TimeUnit.SECONDS).subscribe(new Consumer<Long>() { @Override public void accept(Long aLong) throws Exception { Log.d(TAG, String.valueOf(aLong)); } }); } } 运行结果输出： 03-20 12:46:30.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 1 03-20 12:46:32.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 2 03-20 12:46:34.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 3 03-20 12:46:36.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 4 03-20 12:46:38.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 5 03-20 12:46:40.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 6 03-20 12:46:42.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 7 03-20 12:46:44.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 8 03-20 12:46:46.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 9 03-20 12:46:48.029 4214-4245/? D/55907a88-b6cb-4acb-b2d8-ed9c27cc9003: 10

2017-03-19

区块链的现状与未来

下面是摘自《正在被比特币改变的法律世界》中的一个真实的案例：6月17日星期五，下午快下班的时候，区块链微信群里突然炸开了锅，我于是暂时停下手中的工作，仔细看了一下微信群里大家热议的内容：原来，是以太坊（

2017-03-15

端口映射与容器互联

在实践中，经常会碰到需要多个服务组件容器共同协作的情况，这往往需要多个容器之间能够互相访问到对方的服务。

2017-03-12

Android依赖管理与私服搭建

*本篇文章已授权微信公众号 guolin_blog （郭霖）独家发布 1.Android引入依赖的多种方式 2.多平台私服搭建 3.创建和依赖自己的代码仓库 1、Android引入依赖的多种方式引入方式一

2017-03-03

[Hive]窗口函数与分析函数

具体参考： [Hive]窗口函数LEAD LAG FIRST_VALUE LAST_VALUE 2 OVER子句 OVER子句可以与标准聚合函数使用(COUNT，SUM，MIN，MAX，AVG)。

2017-02-21

docker入门安装与简单使用

（三） Docker 引擎 Docker Engine是一个C/S架构的应用程序，主要有以下组件：服务端守护进程 REST API接口，程序用来与守护进程进行交互 CLI（command line interface

2017-02-16

keepalived高可用LVS与Nginx

notify.shbackup" notify_fault"/etc/keepalived/notify.shfault" } virtual_server192.168.0.8080{ delay_loop6#服务器轮询

2017-02-11

activity 与 service 之间的通信

service: <service android:name="com.chexiu.service.UpdateVersionService" > </service> service与activity

2017-02-10

Java中ArrayDeque，栈与队列

package ch8; import java.util.*; /** * Created by Jiqing on 2016/11/27. */ public class ArrayDequeStack { public static void main(String[] args) { ArrayDeque stack = new ArrayDeque(); // 依次将三个元素push入“栈”，先进后出 stack.push("疯狂Java讲义"); stack.push("轻量级Java EE企业应用实战"); stack.push("疯狂Android讲义"); System.out.println(stack); // [疯狂Android讲义, 轻量级Java EE企业应用实战, 疯狂Java讲义] System.out.println(stack.peek()); // 疯狂Android讲义 System.out.println(stack); // [疯狂Android讲义, 轻量级Java EE企业应用实战, 疯狂Java讲义] System.out.println(stack.pop()); // 疯狂Android讲义 System.out.println(stack);//[轻量级Java EE企业应用实战, 疯狂Java讲义] // 当做队列来使用,先进先出 ArrayDeque queue = new ArrayDeque(); queue.offer("疯狂Java讲义"); queue.offer("轻量级JavaEE企业应用实践"); queue.offer("疯狂Android讲义"); System.out.println(queue); //[疯狂Java讲义, 轻量级JavaEE企业应用实践, 疯狂Android讲义] // 访问队列头部元素，但不将其poll出队列 System.out.println(queue.peek()); System.out.println(queue); // poll出第一个元素 System.out.println(queue.poll()); System.out.println(queue);// [轻量级JavaEE企业应用实践, 疯狂Android讲义] } } 输入结果： [疯狂Android讲义, 轻量级Java EE企业应用实战, 疯狂Java讲义] 疯狂Android讲义 [疯狂Android讲义, 轻量级Java EE企业应用实战, 疯狂Java讲义] 疯狂Android讲义 [轻量级Java EE企业应用实战, 疯狂Java讲义][疯狂Java讲义, 轻量级JavaEE企业应用实践, 疯狂Android讲义] 疯狂Java讲义 [疯狂Java讲义, 轻量级JavaEE企业应用实践, 疯狂Android讲义] 疯狂Java讲义 [轻量级JavaEE企业应用实践, 疯狂Android讲义] 本文转自TBHacker博客园博客，原文链接：http://www.cnblogs.com/jiqing9006/p/6107491.html，如需转载请自行联系原作者

2016-12-31

区块链的现状与展望

区块链与法律这个标题貌似有点大，那我说一个例子：从世界最大众筹项目DAO的技术漏洞引起的以太币被盗风险说起，谈到作为面向未来的法律人应该如何参与到区块链这场世纪浪潮中。

2016-12-10

Flink 原理与实现：Session Window

目前开源领域大部分的流计算引擎都有窗口的概念，但是没有对 session window 的支持，要实现 session window，需要用户自己去做完大部分事情。

2016-11-23

Flink 原理与实现：Window 机制

Flink 认为 Batch 是 Streaming 的一个特例，所以 Flink 底层引擎是一个流式引擎，在上面实现了流处理和批处理。而窗口（window）就是从 Streaming 到 Batch 的一个桥梁。Flink 提供了非常完善的窗口机制，这是我认为的 Flink 最大的亮点之一（其他的亮点包括消息乱序处理，和 checkpoint 机制）。本文我们将介绍流式处理中的窗口概念，介绍 Flink 内建的一些窗口和 Window API，最后讨论下窗口在底层是如何实现的。什么是 Window 在流处理应用中，数据是连续不断的，因此我们不可能等到所有数据都到了才开始处理。当然我们可以每来一个消息就处理一次，但是有时我们需要做一些聚合类的处理，例如：在过去的1分钟内有多少用户点击了我们的网页。在这种情况下，我们必须定义一个窗口，

2016-11-23

IoT技术架构与安全威胁

本文选自《智能硬件安全》一书，将从技术层面向您介绍IoT技术架构与安全威胁。

2016-11-22

阿里云-进程监控与告警

1.背景我在阿里云ECS上部署了个人网站，用到了mysql与tomcat，所以想要分钟级监控这两个进程是否alive。

2016-11-17

Serverless日志处理挑战与方案

本文PPT来自阿里云的简志于10月16日在2016年杭州云栖大会上发表的《Serverless日志处理挑战与方案》。

2016-11-02

[Flume]安装，部署与应用案例

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/SunnyYoona/article/details/52585478 1. 官网 http://flume.apache.org/ 2. 下载 http://flume.apache.org/download.html 3. 安装 3.1 将下载的flume包，解压到/opt目录中 3.2 创建flume-env.sh 配置文件 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume-env.sh.template flume-env.sh 3.3 修改 flume-env.sh 配置文件，主要是JAVA_HOME变量设置 # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced # during Flume startup. # Enviroment variables can be set here. # export JAVA_HOME=/usr/lib/jvm/java-6-sun export JAVA_HOME=/opt/jdk1.8.0_91 # Give Flume more memory and pre-allocate, enable remote monitoring via JMX # export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote" # Note that the Flume conf directory is always included in the classpath. #FLUME_CLASSPATH="" 3.4 验证是否安装成功 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng version Flume 1.6.0 Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080 Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015 From source with checksum b29e416802ce9ece3269d34233baf43f 出现上面信息，表示安装成功了。 4. 案例 4.1 案例一 Avro Avro可以发送一个给定的文件给Flume，Avro 源使用AVRO RPC机制。 4.1.1 创建agent 配置文件根据模板文件创建配置文件： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume-conf.properties.template flume.conf 4.1.2 配置agent配置文件当你运行一个agent的时候，需要通过-f 选项来告诉Flume使用哪个配置文件。让我们看一个基本的例子，复制下面代码并粘贴到conf/flume.conf文件中。 # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = avro agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 4141 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.1.3 启动flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume.conf -n agent1 -Dflume.root.logger=INFO,console 4.1.4 创建指定文件 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ sudo touch log.00 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ sudo vim log.00 4.1.5 使用avro-client发送文件 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng avro-client -c . -H 0.0.0.0 -p 4141 -F ../log.00 4.1.6 查看信息在启动agent的控制窗口，可以看到一下信息，注意最后一行： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 10:29:27 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 10:29:27 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume.conf 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 10:29:27 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 10:29:27 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 10:29:27 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 10:29:27 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 10:29:27 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type avro 16/09/19 10:29:27 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 10:29:27 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 10:29:27 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:Avro source avro-source1: { bindAddress: 0.0.0.0, port: 4141 } }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@453e9d5e counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 10:29:27 INFO node.Application: Starting Channel ch1 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 10:29:27 INFO node.Application: Starting Sink logger-sink1 16/09/19 10:29:27 INFO node.Application: Starting Source avro-source1 16/09/19 10:29:27 INFO source.AvroSource: Starting Avro source avro-source1: { bindAddress: 0.0.0.0, port: 4141 }... 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 10:29:27 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 10:29:27 INFO source.AvroSource: Avro source avro-source1 started. 16/09/19 10:36:32 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 => /127.0.0.1:4141] OPEN 16/09/19 10:36:32 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 => /127.0.0.1:4141] BOUND: /127.0.0.1:4141 16/09/19 10:36:32 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 => /127.0.0.1:4141] CONNECTED: /127.0.0.1:42708 16/09/19 10:36:33 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 :> /127.0.0.1:4141] DISCONNECTED 16/09/19 10:36:33 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 :> /127.0.0.1:4141] UNBOUND 16/09/19 10:36:33 INFO ipc.NettyServer: [id: 0x072a068a, /127.0.0.1:42708 :> /127.0.0.1:4141] CLOSED 16/09/19 10:36:33 INFO ipc.NettyServer: Connection to /127.0.0.1:42708 disconnected. 16/09/19 10:36:37 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 Hello Flume } 4.2 案例二 Spool Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点：（1）拷贝到spool目录下的文件不可以再打开编辑。（2）spool目录下不可包含相应的子目录 4.2.1 创建配置文件flume-spool.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-spool.conf 进行一下配置： # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = spooldir agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.spoolDir = /home/xiaosi/logs/ agent1.sources.avro-source1.fileHeader = true agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 4141 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 对/home/xiaosi/logs目录进行监控。 4.2.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-spool.conf -n agent1 -Dflume.root.logger=INFO,console 4.2.3 追加文件到监控目录 xiaosi@Qunar:~$ echo "Hello Flume first" > /home/xiaosi/logs/flume-log-1.log xiaosi@Qunar:~$ echo "Hello Flume second" > /home/xiaosi/logs/flume-log-2.log 4.2.4 查看信息在启动agent的控制窗口，可以看到一下信息，注意最后两行： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-spool.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Including Hive libraries found via (/opt/apache-hive-2.0.0-bin) for Hive access ... org.apache.flume.node.Application -f ../conf/flume-spool.conf -n agent1 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 11:29:52 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 11:29:52 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-spool.conf 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 11:29:52 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 11:29:52 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 11:29:52 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 11:29:52 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 11:29:52 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type spooldir 16/09/19 11:29:52 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 11:29:52 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 11:29:52 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:Spool Directory source avro-source1: { spoolDir: /home/xiaosi/logs/ } }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4f5f731e counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 11:29:52 INFO node.Application: Starting Channel ch1 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 11:29:52 INFO node.Application: Starting Sink logger-sink1 16/09/19 11:29:52 INFO node.Application: Starting Source avro-source1 16/09/19 11:29:52 INFO source.SpoolDirectorySource: SpoolDirectorySource source starting with directory: /home/xiaosi/logs/ 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 11:29:52 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 11:30:06 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one. 16/09/19 11:30:06 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /home/xiaosi/logs/flume-log-1.log to /home/xiaosi/logs/flume-log-1.log.COMPLETED 16/09/19 11:30:07 INFO sink.LoggerSink: Event: { headers:{file=/home/xiaosi/logs/flume-log-1.log} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 66 69 72 73 Hello Flume firs } 16/09/19 11:30:21 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one. 16/09/19 11:30:21 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /home/xiaosi/logs/flume-log-2.log to /home/xiaosi/logs/flume-log-2.log.COMPLETED 16/09/19 11:30:22 INFO sink.LoggerSink: Event: { headers:{file=/home/xiaosi/logs/flume-log-2.log} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 73 65 63 6F Hello Flume seco } 4.3 案例三 Exec EXEC执行一个给定的命令获得输出的源，如果要使用tail命令，必选使得file足够大才能看到输出内容 4.3.1 创建配置文件flume-exec.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-exec.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = exec agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.command = tail -F /home/xiaosi/logs/flume-log-exec.log agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 4141 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.2.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-exec.conf -n agent1 -Dflume.root.logger=INFO,console 4.2.3 执行tail命令向文件中进行追加数据，生成足够多的数据： #! /bin/sh for index in {1..100} do echo "Hello Flume $index" >> /home/xiaosi/logs/flume-log-exec.log done 同时对文件使用tail 命令操作： xiaosi@Qunar:~$ tail -F /home/xiaosi/logs/flume-log-exec.log Hello Flume 1 Hello Flume 2 Hello Flume 3 Hello Flume 4 ... 4.3.4 查看信息在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-exec.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 12:01:28 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 12:01:28 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-exec.conf 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:01:28 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 12:01:28 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 12:01:28 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 12:01:28 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 12:01:28 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type exec 16/09/19 12:01:28 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 12:01:28 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 12:01:28 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@242d6c8b counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 12:01:28 INFO node.Application: Starting Channel ch1 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 12:01:28 INFO node.Application: Starting Sink logger-sink1 16/09/19 12:01:28 INFO node.Application: Starting Source avro-source1 16/09/19 12:01:28 INFO source.ExecSource: Exec source starting with command:tail -F /home/xiaosi/logs/flume-log-exec.log 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 12:01:28 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 31 Hello Flume 1 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 32 Hello Flume 2 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 33 Hello Flume 3 } ... 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 39 38 Hello Flume 98 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 39 39 Hello Flume 99 } 16/09/19 12:01:58 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 20 31 30 30 Hello Flume 100 } 4.4 案例四 Syslogtcp Syslogtcp监听TCP的端口做为数据源 4.4.1 创建配置文件flume-tcp.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-tcp.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = syslogtcp agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.host = localhost #agent1.sources.avro-source1.bind = 0.0.0.0 agent1.sources.avro-source1.port = 5140 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.4.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-tcp.conf -n agent1 -Dflume.root.logger=INFO,console 4.4.3 测试产生syslog xiaosi@Qunar:~$ echo "hello flume tcp" | nc localhost 5140 4.4.4 查看信息在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-tcp.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 12:10:15 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 12:10:15 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-tcp.conf 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 12:10:15 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 12:10:15 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 12:10:15 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 12:10:15 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 12:10:15 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type syslogtcp 16/09/19 12:10:15 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 12:10:15 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 12:10:15 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogTcpSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@38aab021 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 12:10:15 INFO node.Application: Starting Channel ch1 16/09/19 12:10:16 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 12:10:16 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 12:10:16 INFO node.Application: Starting Sink logger-sink1 16/09/19 12:10:16 INFO node.Application: Starting Source avro-source1 16/09/19 12:10:16 INFO source.SyslogTcpSource: Syslog TCP Source starting... 16/09/19 12:10:50 WARN source.SyslogUtils: Event created from Invalid Syslog data. 16/09/19 12:10:54 INFO sink.LoggerSink: Event: { headers:{Severity=0, Facility=0, flume.syslog.status=Invalid} body: 68 65 6C 6C 6F 20 66 6C 75 6D 65 20 74 63 70 hello flume tcp } 4.5 案例五 JSONHandler 4.5.1 创建配置文件flume-json.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-json.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = org.apache.flume.source.http.HTTPSource agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.port = 8888 # sink agent1.sinks.logger-sink1.type = logger agent1.sinks.logger-sink1.channel = ch1 # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.5.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-json.conf -n agent1 -Dflume.root.logger=INFO,console 4.5.3 生成JSON 格式的POST request xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ curl -X POST -d '[{ "headers" :{"a":"a1", "b":"b1"}, "body":"flume_json_boy"}]' http://localhost:8888 4.5.4 查看信息在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-json.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 13:21:28 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 13:21:28 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-json.conf 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 13:21:28 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 13:21:28 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 13:21:28 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 13:21:28 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 13:21:28 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type org.apache.flume.source.http.HTTPSource 16/09/19 13:21:28 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: logger 16/09/19 13:21:28 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 13:21:28 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@136bcdd0 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 13:21:28 INFO node.Application: Starting Channel ch1 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 13:21:28 INFO node.Application: Starting Sink logger-sink1 16/09/19 13:21:28 INFO node.Application: Starting Source avro-source1 16/09/19 13:21:28 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 16/09/19 13:21:28 INFO mortbay.log: jetty-6.1.26 16/09/19 13:21:28 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:8888 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean. 16/09/19 13:21:28 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-source1 started 16/09/19 13:21:32 INFO sink.LoggerSink: Event: { headers:{a=a1, b=b1} body: 66 6C 75 6D 65 5F 6A 73 6F 6E 5F 62 6F 79 flume_json_boy } 4.6 案例六 Hadoop Sink Syslogtcp监听TCP的端口做为数据源，并将监听的数据存储在HDFS中 4.6.1 创建配置文件flume-hadoop.conf xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/conf$ sudo cp flume.conf flume-hadoop.conf 进行如下修改： # The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called 'agent' agent1.sources = avro-source1 agent1.channels = ch1 agent1.sinks = logger-sink1 # sources agent1.sources.avro-source1.type = syslogtcp agent1.sources.avro-source1.channels = ch1 agent1.sources.avro-source1.host = localhost agent1.sources.avro-source1.port = 5140 # sink agent1.sinks.logger-sink1.type = hdfs agent1.sinks.logger-sink1.channel = ch1 agent1.sinks.logger-sink1.hdfs.path = hdfs://localhost:9000/user/xiaosi/data agent1.sinks.logger-sink1.hdfs.filePrefix = SysLog agent1.sinks.logger-sink1.hdfs.round = true agent1.sinks.logger-sink1.hdfs.roundValue = 10 agent1.sinks.logger-sink1.hdfs.roundUnit = minute # channel agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 1000 agent1.channels.ch1.transactionCapacity = 100 4.6.2 启动Flume agent agent1 xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-hadoop.conf -n agent1 -Dflume.root.logger=INFO,console 4.6.3 测试产生syslog xiaosi@Qunar:/opt/apache-flume-1.6.0-bin$ echo "Hello Flume -> Hadoop one" | nc localhost 5140 4.6.4 查看信息在启动agent的控制窗口，可以看到一下信息： xiaosi@Qunar:/opt/apache-flume-1.6.0-bin/bin$ flume-ng agent -c . -f ../conf/flume-hadoop.conf -n agent1 -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/opt/hadoop-2.7.2/bin/hadoop) for HDFS access Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath ... SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-flume-1.6.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/09/19 13:34:58 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/09/19 13:34:58 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:../conf/flume-hadoop.conf 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Added sinks: logger-sink1 Agent: agent1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Processing:logger-sink1 16/09/19 13:34:58 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1] 16/09/19 13:34:58 INFO node.AbstractConfigurationProvider: Creating channels 16/09/19 13:34:58 INFO channel.DefaultChannelFactory: Creating instance of channel ch1 type memory 16/09/19 13:34:58 INFO node.AbstractConfigurationProvider: Created channel ch1 16/09/19 13:34:58 INFO source.DefaultSourceFactory: Creating instance of source avro-source1, type syslogtcp 16/09/19 13:34:58 INFO sink.DefaultSinkFactory: Creating instance of sink: logger-sink1, type: hdfs 16/09/19 13:34:58 INFO node.AbstractConfigurationProvider: Channel ch1 connected to [avro-source1, logger-sink1] 16/09/19 13:34:58 INFO node.Application: Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:org.apache.flume.source.SyslogTcpSource{name:avro-source1,state:IDLE} }} sinkRunners:{logger-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@569671b3 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 16/09/19 13:34:58 INFO node.Application: Starting Channel ch1 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started 16/09/19 13:34:58 INFO node.Application: Starting Sink logger-sink1 16/09/19 13:34:58 INFO node.Application: Starting Source avro-source1 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: logger-sink1: Successfully registered new MBean. 16/09/19 13:34:58 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: logger-sink1 started 16/09/19 13:34:58 INFO source.SyslogTcpSource: Syslog TCP Source starting... 16/09/19 13:35:06 WARN source.SyslogUtils: Event created from Invalid Syslog data. 16/09/19 13:35:07 INFO hdfs.HDFSSequenceFile: writeFormat = Writable, UseRawLocalFileSystem = false 16/09/19 13:35:07 INFO hdfs.BucketWriter: Creating hdfs://localhost:9000/user/xiaosi/data/SysLog.1474263307767.tmp 4.6.5 查看HDFS xiaosi@Qunar:/opt/hadoop-2.7.2/sbin$ hadoop fs -ls /user/xiaosi/data Found 3 items -rw-r--r-- 1 xiaosi supergroup 141 2016-09-19 13:35 /user/xiaosi/data/SysLog.1474263307767 -rw-r--r-- 1 xiaosi supergroup 1350 2016-07-28 14:10 /user/xiaosi/data/mysql-result.txt -rw-r--r-- 3 xiaosi supergroup 26 2016-07-30 22:47 /user/xiaosi/data/num.txt xiaosi@Qunar:/opt/hadoop-2.7.2/sbin$ hadoop fs -text /user/xiaosi/data/SysLog.1474263307767 1474263309104 48 65 6c 6c 6f 20 46 6c 75 6d 65 20 2d 3e 20 48 61 64 6f 6f 70 20 20 6f 6e 65 xiaosi@Qunar:/opt/hadoop-2.7.2/sbin$ hadoop fs -cat /user/xiaosi/data/SysLog.1474263307767 SEQ

2016-09-18

RecyclerView与SwipeRefreshLayout的完美搭配

SwipeRefreshLayout 是谷歌公司推出的用于下拉刷新的控件，在Version 19.1之后SwipeRefreshLayout被放到support v4中。我们在前面也已经说过一个SwipeRefreshLayout和ListView实现上拉加载、下拉刷新的例子。而现在RecyclerView也已作为实现列表功能的首选组件，而它本身并没有实现上拉加载、下拉刷新的功能，本节我们就来利用SwipeRefreshLayout 和RecyclerView来实现上拉加载、下拉刷新的功能。 SwipeRefreshLayout 常用的方法如下: setOnRefreshListener(OnRefreshListener):下拉刷新监听器，当用户下拉的时候会触发 setRefreshing(boolean):设置SwipeRefreshLayout当前是否处于刷新状态，一般是在请求数据的时候设置为true，在数据被加载到View中后，设置为false。 isRefreshing():检查是否处于刷新状态 setColorSchemeResources():设置进度条的颜色主题，最多设置四种，以前的setColorScheme()方法已经弃用了。 setProgressViewOffset(boolean scale, int start, int end) 调整进度条距离屏幕顶部的距离布局文件很简单 <?xml version="1.0" encoding="utf-8"?> <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" android:paddingBottom="@dimen/activity_vertical_margin" android:paddingLeft="@dimen/activity_horizontal_margin" android:paddingRight="@dimen/activity_horizontal_margin" android:paddingTop="@dimen/activity_vertical_margin" tools:context="com.xmr.recycleandswiperefresh.MainActivity"> <android.support.v4.widget.SwipeRefreshLayout android:id="@+id/srl_main" android:layout_width="match_parent" android:layout_height="match_parent"> <android.support.v7.widget.RecyclerView android:id="@+id/ryv_main" android:layout_width="match_parent" android:layout_height="match_parent" /> </android.support.v4.widget.SwipeRefreshLayout> </RelativeLayout> 接着在Activity中获取SwipeRefreshLayout控件并且设置OnRefreshListener监听器，同时实现里边的onRefresh()方法，在该方法中进行网络请求最新数据，然后刷新RecyclerView列表同时设置SwipeRefreshLayout的显示效果。具体代码如下： mSwipeRefreshLayout.setOnRefreshListener(new SwipeRefreshLayout.OnRefreshListener() { @Override public void onRefresh() { //实际项目中这里一般是用网络请求获取数据 mList.add("下拉刷新"); mAdapter.notifyDataSetChanged(); mSwipeRefreshLayout.setRefreshing(false); } }); RecyclerView的实现：第一种，下拉刷新和上拉刷新都用SwipeRefreshLayout自带的效果 mRecyclerView.setOnScrollListener(new RecyclerView.OnScrollListener() { @Override public void onScrollStateChanged(RecyclerView recyclerView, int newState) { super.onScrollStateChanged(recyclerView, newState); if (newState == RecyclerView.SCROLL_STATE_IDLE && lastVisibleItem + 1 == mAdapter.getItemCount()) { mSwipeRefreshLayout.setRefreshing(true); //实际项目中这里一般是用网络请求获取数据 mList.add("上拉加载"); //为了有刷新的效果，延迟关闭刷新效果 mSwipeRefreshLayout.postDelayed(new Runnable() { @Override public void run() { mSwipeRefreshLayout.setRefreshing(false); mAdapter.notifyDataSetChanged(); } }, 2000); } } @Override public void onScrolled(RecyclerView recyclerView, int dx, int dy) { super.onScrolled(recyclerView, dx, dy); //判断是当前layoutManager是否为LinearLayoutManager // 只有LinearLayoutManager才有查找第一个和最后一个可见view位置的方法 if (linearLayoutManager instanceof LinearLayoutManager) { //获取最后一个可见view的位置 lastVisibleItem = linearLayoutManager.findLastVisibleItemPosition(); //获取第一个可见view的位置 // int firstItemPosition = linearLayoutManager.findFirstVisibleItemPosition(); } } }); 效果如下：第二种实现下拉刷新用SwipeRefreshLayout自带的效果，上拉加载我们可以给RecyclerView 添加一个类似FooterView的item。我们在Adapter中实现： package com.xmr.recycleandswiperefresh; import android.content.Context; import android.support.v7.widget.RecyclerView; import android.view.LayoutInflater; import android.view.View; import android.view.ViewGroup; import android.widget.LinearLayout; import android.widget.TextView; import java.util.List; /** * Created by Administrator on 2016/9/13. */ public class RefreshAdapter1 extends RecyclerView.Adapter<RecyclerView.ViewHolder> { private Context mContext; private List<String> mDatas; private static final int TYPE_ITEM = 0; //普通Item View private static final int TYPE_FOOTER = 1; //底部FootView //已经到底了 public static final int PULLUP_LOAD_MORE = 0; //正在加载中 public static final int LOADING_MORE = 1; //上拉加载状态-默认为0 private int load_more_status = 0; public RefreshAdapter1(Context mContext, List<String> mDatas) { this.mContext = mContext; this.mDatas = mDatas; } @Override public RecyclerView.ViewHolder onCreateViewHolder(ViewGroup parent, int viewType) { //进行判断显示类型，来创建返回不同的View if (viewType == TYPE_ITEM) { View view = LayoutInflater.from(parent.getContext()).inflate( R.layout.item_recycler_refresh, null); // view.setLayoutParams(new LayoutParams(LayoutParams.MATCH_PARENT, // LayoutParams.WRAP_CONTENT)); return new ItemHolder(view); } // type == TYPE_FOOTER 返回footerView else if (viewType == TYPE_FOOTER) { View view = LayoutInflater.from(parent.getContext()).inflate( R.layout.item_recycler_footer, null); // view.setLayoutParams(new LayoutParams(LayoutParams.MATCH_PARENT, // LayoutParams.WRAP_CONTENT)); return new FooterHolder(view); } return null; } @Override public void onBindViewHolder(RecyclerView.ViewHolder holder, int position) { if (holder instanceof ItemHolder) { ItemHolder itemHolder = (ItemHolder) holder; itemHolder.textView.setText(mDatas.get(position)); } else if (holder instanceof FooterHolder) { FooterHolder footHolder = (FooterHolder) holder; //这句如果不注释，那么底部就会有添加的一个item footHolder.linearLayout.setVisibility(View.GONE); switch (load_more_status) { case PULLUP_LOAD_MORE: footHolder.linearLayout.setVisibility(View.GONE); // footHolder.textView.setText("已经到底了"); break; case LOADING_MORE: footHolder.linearLayout.setVisibility(View.VISIBLE); footHolder.textView.setText("正在加载更多数据..."); break; } } } @Override public int getItemCount() { return mDatas == null ? 0 : mDatas.size() + 1; } @Override public int getItemViewType(int position) { // 最后一个item设置为footerView if (position + 1 == getItemCount()) { return TYPE_FOOTER; } else { return TYPE_ITEM; } } class ItemHolder extends RecyclerView.ViewHolder { public TextView textView; public ItemHolder(View itemView) { super(itemView); textView = (TextView) itemView.findViewById(R.id.tv_item_recycler_refresh); } } class FooterHolder extends RecyclerView.ViewHolder { public TextView textView; public LinearLayout linearLayout; public FooterHolder(View itemView) { super(itemView); textView = (TextView) itemView.findViewById(R.id.tv_item_recycler_footer); linearLayout = (LinearLayout) itemView.findViewById(R.id.ll_item_recycler_footer); } } /** * //上拉加载更多 * PULLUP_LOAD_MORE=0; * //正在加载中 * LOADING_MORE=1; * //加载完成已经没有更多数据了 * NO_MORE_DATA=2; * * @param status */ public void changeMoreStatus(int status) { load_more_status = status; notifyDataSetChanged(); } } 而对RecyclerView的上拉加载，我们这样实现： mRecyclerView.setOnScrollListener(new RecyclerView.OnScrollListener() { @Override public void onScrollStateChanged(RecyclerView recyclerView, int newState) { super.onScrollStateChanged(recyclerView, newState); if (newState == RecyclerView.SCROLL_STATE_IDLE && lastVisibleItem + 1 == mAdapter.getItemCount()) { Log.i(">>>>>", ">>>>" + lastVisibleItem + ">>>" + mAdapter.getItemCount()); mAdapter.changeMoreStatus(mAdapter.LOADING_MORE); //为了有刷新的效果，延迟2s修改状态 new Handler().postDelayed(new Runnable() { @Override public void run() { mAdapter.changeMoreStatus(RefreshAdapter1.PULLUP_LOAD_MORE); //实际项目中这里一般是用网络请求获取数据 mList.add("上拉加载"); } }, 2000); } } @Override public void onScrolled(RecyclerView recyclerView, int dx, int dy) { super.onScrolled(recyclerView, dx, dy); //判断是当前layoutManager是否为LinearLayoutManager // 只有LinearLayoutManager才有查找第一个和最后一个可见view位置的方法 if (linearLayoutManager instanceof LinearLayoutManager) { //获取最后一个可见view的位置 lastVisibleItem = linearLayoutManager.findLastVisibleItemPosition(); //获取第一个可见view的位置 // int firstItemPosition = linearLayoutManager.findFirstVisibleItemPosition(); } } }); } 最终效果如下：不过，现在也有点bug，如果列表没有占满屏幕的话，下拉刷新的时候也会同时触发上拉加载的效果，大家如果有好的解决方案的话可以给我说下，先谢谢了参考： http://blog.csdn.net/dalancon/article/details/46125667 http://www.cnblogs.com/xiaoyaoxia/p/4977125.html http://blog.csdn.net/developer_jiangqq/article/details/49992269 http://blog.csdn.net/salute_li/article/details/52118504 http://www.cnblogs.com/csonezp/p/5050324.html

2016-09-13

Spark Streaming 的玫瑰与刺

前言说人话：其实就是讲Spark Streaming 的好处与坑。好处主要从一些大的方面讲，坑则是从实际场景中遇到的一些小细节描述。玫瑰篇玫瑰篇主要是说Spark Streaming的优势点。

2016-09-07

iOS - OC 与 Swift 互相操作

前言在 Swift 语言中，我们可以使用 Objective-C、C 语言编写代码，我们可以导入任意用 Objective-C 写的 Cocoa 平台框架、Objective-C 框架或 C 类库。 1、在 Swift 中使用 Objective-C、C 代码 1.1 在 Swift 项目中使用 Objective-C、C 代码 1> 在 Swift 项目中创建 OC/C 的 .h 和 .m / .c 文件，文件中的格式为其本有的格式。 2> Xcode 提示是否创建 Objective-C bridging 头文件，点击 Yes 创建，Xcode 会自动创建一个桥接头文件，文件名的格式是 "项目名-Bridging-Header.h"。一般情况下，Xcode 辅助生成的桥接头文件在编译属性里已经帮我们设置了引用，如果没有的话需要自己将其配置到项目中，选中项目中的 Targets => Build Setting => Swift Compiler - general => Objective-C Bridging Header 属性，双击配置成 “项目名/项目名-Bridging-Header.h” 即可。 3> 在此桥接头文件中添加所创建的 OC/C 文件的头文件（ #import "*****.h" ）。 4> 在 main.swift 文件中无需添加头文件，直接使用创建的 OC/C 文件中的类、方法或函数即可。 1.2 在 Swift 项目中的 Objective-C 代码中使用 Swift 代码 1> 在 .Swift 文件中每一个方法和类名前都加 @objc 前缀来修饰，以便在 Objective-C 中可以访问。新版 Swift 代码中已不需要再添加。 2> 在 Swift 项目中创建 OC 的 .h 和 .m 文件，文件中的格式为其本有的格式。 3> Xcode 提示是否创建 Objective-C bridging 头文件，点击 Yes 创建，Xcode 会自动创建一个桥接头文件，文件名的格式是 "项目名-Bridging-Header.h"。一般情况下，Xcode 辅助生成的桥接头文件在编译属性里已经帮我们设置了引用，如果没有的话需要自己将其配置到项目中，选中项目中的 Targets => Build Setting => Swift Compiler - general => Objective-C Bridging Header 属性，双击配置成 “项目名/项目名-Bridging-Header.h” 即可。 4> 在此桥接头文件中添加所创建的 OC 文件的头文件（ #import "*****.h" ）。 5> 在 OC 的 .m 文件中添加头文件 “#import "项目名-Swift.h"”，在此文件中直接使用 .Swift 文件中的类、方法即可。 "项目名-Swift.h" 这个头文件是 Xcode 自动产生的，如果不是引用外部 Swift 语言编写的框架，不需要我们做过多配置，但是如果是引用外部框架，需要我们进行配置：选中项目中的 Targets => Build Setting => Packaging => Defines Module 修改成 Yes，然后在 Build Setting => Packaging => Product Module Name，设置一个名称，例如 SwiftInAction。特别注意的是，这个名称不能有 “-” 等字符。这样我们在 OC 的 .m 文件中就可以添加 "SwiftInAction-Swift.h" 了，这个名字是 Xcode 根据设置的模块名自动生成的。 6> 在 main.swift 文件中无需添加头文件，直接使用创建的 OC 文件中的类、方法。 2、在 Objective-C 中使用 Swift 代码 2.1 在 Objective-C 项目中使用 Swift 代码 1、在 OC 项目中创建 .Swift 文件，文件中的格式为其本有的格式。 2、Xcode 提示是否创建 Objective-C bridging 头文件，点击 Yes 创建，Xcode 会自动创建一个桥接头文件，文件名的格式是 "项目名-Bridging-Header.h"。一般情况下，Xcode 辅助生成的桥接头文件在编译属性里已经帮我们设置了引用，如果没有的话需要自己将其配置到项目中，选中项目中的 Targets => Build Setting => Swift Compiler - general => Objective-C Bridging Header 属性，双击配置成 “项目名/项目名-Bridging-Header.h” 即可。 3、在 OC 的 .m 文件中添加头文件 “#import "项目名-Swift.h"”，在此文件中直接使用 .Swift 文件中的类、方法即可。 "项目名-Swift.h" 这个头文件是 Xcode 自动产生的，如果不是引用外部 Swift 语言编写的框架，不需要我们做过多配置，但是如果是引用外部框架，需要我们进行配置：选中项目中的 Targets => Build Setting => Packaging => Defines Module 修改成 Yes，然后在 Build Setting => Packaging => Product Module Name，设置一个名称，例如 SwiftInAction。特别注意的是，这个名称不能有 “-” 等字符。这样我们在 OC 的 .m 文件中就可以添加 "SwiftInAction-Swift.h" 了，这个名字是 Xcode 根据设置的模块名自动生成的。在开始的几次编译或者代码调试中，可能会出错，提示找不到这个自动生成的文件，一般对项目进行 clean 操作可以解决问题，具体是点击 Product 菜单下的 clean 菜单项即可。 2.2 在 Objective-C 里使用 Swift 限制 1、一些 Swift 特有的类型，例如元组、泛型、类型别名、嵌套类型等，不能在 Objective-C 里使用。 2、不能在 Objective-C 里继承 Swift 类型。 3、数据类型之间需要互相映射关联。可以 Objective-C 核心类型到 Swift 中的替代类型，例如 NSString 到 String。Objective-C 确定类型到 Swift 中的同等类型，例如 id 到 AnyObject。Objective-C 概念到 Swift 中相匹配的概念，如 pointers 到 optionals。 4、不能直接把 C++ 代码导入 Swift，解决办法是为 C++ 代码创建一个 Objective-C 或者 C 的封装。

2016-08-16

精选列表