首页 文章 精选 留言 我的

精选列表

搜索[API集成]，共10000篇文章
优秀的个人博客,低调大师

Android 3.0 r1 API中文文档(108) —— ExpandableListAdapter

正文 一、结构 public interfaceExpandableListAdapter android.widget.ExpandableListAdapter 间接子类 BaseExpandableListAdapter,CursorTreeAdapter,ResourceCursorTreeAdapter, SimpleCursorTreeAdapter, SimpleExpandableListAdapter 二、概述 这个适配器在 ExpandableListView 和底层数据之间起到了一个衔接的作用。该接口的实现类提供了访问子元素（以组的形式将它们分类）的数据；同样，也提供了为子元素和组创建相应的视图。 三、公共方法 public abstract booleanareAllItemsEnabled() ExpandableListAdapter里面的所有条目都可用吗？如果是yes，就意味着所有条目可以选择和点击了。 返回值 返回True表示所有条目均可用。 参见 areAllItemsEnabled() public abstract CursorgetChild(int groupPosition, int childPosition) 获取指定组中的指定子元素数据。 参数 groupPosition组位置（该组内部含有子元素） childPosition子元素位置（相对于其它子元素） 返回值 返回指定子元素数据。 public abstract longgetChildId(int groupPosition, int childPosition) 获取指定组中的指定子元素ID，这个ID在组里一定是唯一的。联合ID（参见getCombinedChildId(long, long)）在所有条目（所有组和所有元素）中也是唯一的。 参数 groupPosition组位置（该组内部含有子元素） childPosition子元素位置（相对于其它子元素） 返回值 子元素关联ID。 public abstract ViewgetChildView(int groupPosition, int childPosition, boolean isLastChild, View convertView, ViewGroup parent) 获取一个视图对象，显示指定组中的指定子元素数据。 参数 groupPosition组位置（该组内部含有子元素） childPosition子元素位置（决定返回哪个视图） isLastChild子元素是否处于组中的最后一个 convertView重用已有的视图(View)对象。注意：在使用前你应该检查一下这个视图对象是否非空并且这个对象的类型是否合适。由此引伸出，如果该对象不能被转换并显示正确的数据，这个方法就会调用getChildView(int, int, boolean, View, ViewGroup)来创建一个视图(View)对象。 parent 返回的视图(View)对象始终依附于的视图组。 返回值 指定位置上的子元素返回的视图对象 public abstract intgetChildrenCount(int groupPosition) 获取指定组中的子元素个数 参数 groupPosition组位置（决定返回哪个组的子元素个数） 返回值 指定组的子元素个数 public abstract longgetCombinedChildId(long groupId, long childId) 从列表所有项(组或子项)中获得一个唯一的子ID号。可折叠列表要求每个元素(组或子项)在所有的子元素和组中有一个唯一的ID。本方法负责根据所给的子ID号和组ID号返回唯一的ID。此外，若hasStableIds()是true，那么必须要返回稳定的ID。 参数 groupId包含该子元素的组ID childId子元素的ID 返回值 列表所有项(组或子项)中唯一的(和可能稳定)的子元素ID号。（译者注：ID理论上是稳定的，不会发生冲突的情况。也就是说，这个列表会有组、子元素，它们的ID都是唯一的。） public abstract CursorgetGroup(int groupPosition) 获取指定组中的数据 参数 groupPosition组位置 返回值 返回组中的数据，也就是该组中的子元素数据。 public abstract intgetGroupCount() 获取组的个数 返回值 组的个数 public abstract longgetGroupId(int groupPosition) 获取指定组的ID，这个组ID必须是唯一的。联合ID(参见getCombinedGroupId(long))在所有条目(所有组和所有元素)中也是唯一的。 参数 groupPosition组位置 返回值 返回组相关ID public abstract ViewgetGroupView(int groupPosition, boolean isExpanded, View convertView, ViewGroup parent) 获取显示指定组的视图对象。这个方法仅返回关于组的视图对象，要想获取子元素的视图对象，就需要调用getChildView(int, int, boolean, View, ViewGroup)。 参数 groupPosition组位置（决定返回哪个视图） isExpanded 该组是展开状态还是伸缩状态 convertView重用已有的视图对象。注意：在使用前你应该检查一下这个视图对象是否非空并且这个对象的类型是否合适。由此引伸出，如果该对象不能被转换并显示正确的数据，这个方法就会调用getGroupView(int, boolean, View, ViewGroup)来创建一个视图(View)对象。 parent返回的视图对象始终依附于的视图组。 返回值 返回指定组的视图对象 public abstract booleanhasStableIds() 组和子元素是否持有稳定的ID,也就是底层数据的改变不会影响到它们。 返回值 返回一个Boolean类型的值，如果为TRUE，意味着相同的ID永远引用相同的对象。 public abstract booleanisChildSelectable(int groupPosition, int childPosition) 是否选中指定位置上的子元素。 参数 groupPosition组位置（该组内部含有这个子元素） childPosition子元素位置 返回值 是否选中子元素 public abstract booleanisEmpty() 返回值 如果当前适配器不包含任何数据则返回True。经常用来决定一个空视图是否应该被显示。一个典型的实现将返回表达式getCount() == 0的结果，但是由于getCount()包含了头部和尾部，适配器可能需要不同的行为。 参见 isEmpty() public abstract voidonGroupCollapsed(int groupPosition) 当组收缩状态的时候此方法被调用。 参数 groupPosition收缩状态的组索引 public abstract voidonGroupExpanded(int groupPosition) 当组展开状态的时候此方法被调用。 参数 groupPosition展开状态的组位置 public abstract voidregisterDataSetObserver(DataSetObserver observer) 注册一个观察者(observer)，当此适配器数据修改时即调用此观察者。 参数 observer当数据修改时通知调用的对象。 public abstract voidunregisterDataSetObserver(DataSetObserver observer) 取消先前通过registerDataSetObserver(DataSetObserver)方式注册进该适配器中的观察者对象。 参数 observer 取消这个观察者的注册 四、补充 文章精选 android可展开（收缩）的列表ListView(ExpandableListView) android CursorAdapter的监听事件 ExpandableListView.OnChildClickListener 译者署名：情敌贝多芬 译者链接：http://liubey.javaeye.com/ 版本：Android 3.0 r1 结构 继承关系 public static interfaceExpandableListView.OnChildClickListener java.lang.Object android.widget.ExpandableListView.OnChildClickListener 子类及间接子类 间接子类 ExpandableListActivity 类概述 这是一个定义了当可折叠列表(expandable list)里的子元素(child)发生点击事件时调用的回调方法的接口。 公共方法 public abstract booleanonChildClick(ExpandableListView parent, View v, int groupPosition, int childPosition, long id) 用当可折叠列表里的子元素(child)被点击的时候被调用的回调方法。 参数 parent 发生点击动作的ExpandableListView v 在expandable list/ListView中被点击的视图(View) groupPosition 包含被点击子元素的组(group)在ExpandableListView中的位置(索引) childPosition被点击子元素(child)在组(group)中的位置 id 被点击子元素(child)的行ID(索引) 返回值 当点击事件被处理时返回 true ExpandableListView.OnGroupClickListener 译者署名：情敌贝多芬 译者链接：http://liubey.javaeye.com/ 版本：Android 3.0 r1 结构 继承关系 public static interfaceExpandableListView.OnGroupClickListener java.lang.Object android.widget.ExpandableListView.OnGroupClickListener 类概述 这是一个定义了当可折叠列表(expandable list)里的组(group)发生点击事件时调用的回调方法的接口。 公共方法 public abstract booleanonGroupClick(ExpandableListView parent, View v, int groupPosition, long id) 用当可折叠列表里的组(group)被点击的时候被调用的回调方法。 参数 parent发生点击事件的ExpandableListConnector v 在expandable list/ListView中被点击的视图(View) groupPosition被点击的组(group)在ExpandableListConnector中的位置(索引) id 被点击的组(group)的行ID(索引) 返回值 当点击事件被处理的时候返回true ExpandableListView.OnGroupCollapseListener 译者署名：深夜未眠 译者链接：http://chris1012f.javaeye.com/ 版本：Android 3.0 r1 结构 继承关系 public interfaceExpandableListView.OnGroupCollapseListener java.lang.Object android.widget.ExpandableListView.OnGroupCollapseListener 子类及间接子类 间接子类 ExpandableListActivity 类概述 当收缩某个组时，就会发出通知。 公共方法 public abstract void onGroupCollapse (int groupPosition) 每当收缩当前可伸缩列表中的某个组时，就调用该方法。 参数 groupPosition组位置，也就是收缩的那个组的位置。 ExpandableListView.OnGroupExpandListener 译者署名：深夜未眠 译者链接：http://chris1012f.javaeye.com/ 版本：Android 3.0 r1 结构 继承关系 public interfaceExpandableListView.OnGroupExpandListener java.lang.Object android.widget.ExpandableListView.OnGroupExpandListener 子类及间接子类 间接子类 ExpandableListActivity 类概述 当展开某个组时，就会发出通知。 公共方法 public abstract void onGroupExpand (int groupPosition) 每当展开当前可伸缩列表中的某个组时，就调用该方法。 参数 groupPosition 组位置，也就是展开的那个组的位置。 本文转自over140 51CTO博客，原文链接：http://blog.51cto.com/over140/582376，如需转载请自行联系原作者
2017-11-13
优秀的个人博客,低调大师

Hadoop MapReduce编程 API入门系列之最短路径（十五）

====================================== = Iteration: 1 = Input path: out/shortestpath/input.txt = Output path: out/shortestpath/1 ====================================== 2016-12-12 16:37:05,638 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Initializing JVM Metrics with processName=JobTracker, sessionId= 2016-12-12 16:37:06,231 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-12 16:37:06,236 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-12 16:37:06,260 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-12 16:37:06,363 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-12 16:37:06,831 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local535100118_0001 2016-12-12 16:37:07,524 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-12 16:37:07,526 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local535100118_0001 2016-12-12 16:37:07,534 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-12 16:37:07,550 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-12 16:37:07,635 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-12 16:37:07,638 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local535100118_0001_m_000000_0 2016-12-12 16:37:07,716 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 16:37:07,759 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@27b70923 2016-12-12 16:37:07,767 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/shortestpath/input.txt:0+149 2016-12-12 16:37:07,830 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-12 16:37:07,830 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-12 16:37:07,830 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-12 16:37:07,830 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-12 16:37:07,830 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-12 16:37:07,834 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer input -> K[dee],V[0 null hadoop hello] output -> K[dee],V[0 hadoop hello] output -> K[hadoop],V[1 dee] output -> K[hello],V[1 dee] input -> K[hadoop],V[2147483647 null hive hello] output -> K[hadoop],V[2147483647 hive hello] input -> K[hello],V[2147483647 null dee hadoop hive joe] output -> K[hello],V[2147483647 dee hadoop hive joe] input -> K[hive],V[2147483647 null hadoop hello joe] output -> K[hive],V[2147483647 hadoop hello joe] input -> K[joe],V[2147483647 null hive hello] output -> K[joe],V[2147483647 hive hello] 2016-12-12 16:37:07,851 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-12 16:37:07,851 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-12 16:37:07,851 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-12 16:37:07,851 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 174; bufvoid = 104857600 2016-12-12 16:37:07,852 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214372(104857488); length = 25/6553600 2016-12-12 16:37:07,871 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-12 16:37:07,877 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local535100118_0001_m_000000_0 is done. And is in the process of committing 2016-12-12 16:37:07,891 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/shortestpath/input.txt:0+149 2016-12-12 16:37:07,892 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local535100118_0001_m_000000_0' done. 2016-12-12 16:37:07,892 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local535100118_0001_m_000000_0 2016-12-12 16:37:07,892 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-12 16:37:07,896 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-12 16:37:07,896 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local535100118_0001_r_000000_0 2016-12-12 16:37:07,910 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 16:37:07,942 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@5bf7b707 2016-12-12 16:37:07,948 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@969f4cd 2016-12-12 16:37:07,972 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-12 16:37:07,975 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local535100118_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-12 16:37:08,017 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#1 about to shuffle output of map attempt_local535100118_0001_m_000000_0 decomp: 190 len: 194 to MEMORY 2016-12-12 16:37:08,023 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 190 bytes from map-output for attempt_local535100118_0001_m_000000_0 2016-12-12 16:37:08,076 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 190, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->190 2016-12-12 16:37:08,078 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-12 16:37:08,080 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 16:37:08,081 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-12 16:37:08,110 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 16:37:08,111 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 184 bytes 2016-12-12 16:37:08,113 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 190 bytes to disk to satisfy reduce memory limit 2016-12-12 16:37:08,114 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 194 bytes from disk 2016-12-12 16:37:08,115 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-12 16:37:08,116 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 16:37:08,117 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 184 bytes 2016-12-12 16:37:08,118 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 16:37:08,141 INFO [org.apache.hadoop.conf.Configuration.deprecation] - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords input -> K[dee] input -> V[0 hadoop hello] output -> K[dee],V[0 null hadoop hello] input -> K[hadoop] input -> V[2147483647 hive hello] input -> V[1 dee] output -> K[hadoop],V[1 dee hive hello] input -> K[hello] input -> V[2147483647 dee hadoop hive joe] input -> V[1 dee] output -> K[hello],V[1 dee dee hadoop hive joe] input -> K[hive] input -> V[2147483647 hadoop hello joe] output -> K[hive],V[2147483647 null hadoop hello joe] input -> K[joe] input -> V[2147483647 hive hello] output -> K[joe],V[2147483647 null hive hello] 2016-12-12 16:37:08,154 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local535100118_0001_r_000000_0 is done. And is in the process of committing 2016-12-12 16:37:08,156 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 16:37:08,156 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local535100118_0001_r_000000_0 is allowed to commit now 2016-12-12 16:37:08,162 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local535100118_0001_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/shortestpath/1/_temporary/0/task_local535100118_0001_r_000000 2016-12-12 16:37:08,163 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-12 16:37:08,164 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local535100118_0001_r_000000_0' done. 2016-12-12 16:37:08,164 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local535100118_0001_r_000000_0 2016-12-12 16:37:08,164 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-12 16:37:08,535 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local535100118_0001 running in uber mode : false 2016-12-12 16:37:08,539 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-12 16:37:08,544 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local535100118_0001 completed successfully 2016-12-12 16:37:08,601 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 33 File System Counters FILE: Number of bytes read=1340 FILE: Number of bytes written=387869 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=5 Map output records=7 Map output bytes=174 Map output materialized bytes=194 Input split bytes=135 Combine input records=0 Combine output records=0 Reduce input groups=5 Reduce shuffle bytes=194 Reduce input records=7 Reduce output records=5 Spilled Records=14 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=466616320 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=169 File Output Format Counters Bytes Written=161 ====================================== = Iteration: 2 = Input path: out/shortestpath/1 = Output path: out/shortestpath/2 ====================================== 2016-12-12 16:37:08,638 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-12 16:37:08,649 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-12 16:37:08,653 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-12 16:37:09,079 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-12 16:37:09,098 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-12 16:37:09,183 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local447108750_0002 2016-12-12 16:37:09,525 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-12 16:37:09,525 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local447108750_0002 2016-12-12 16:37:09,527 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-12 16:37:09,529 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-12 16:37:09,540 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-12 16:37:09,540 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local447108750_0002_m_000000_0 2016-12-12 16:37:09,544 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 16:37:09,591 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@25a02403 2016-12-12 16:37:09,597 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/shortestpath/1/part-r-00000:0+149 2016-12-12 16:37:09,662 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-12 16:37:09,663 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-12 16:37:09,663 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-12 16:37:09,663 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-12 16:37:09,663 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-12 16:37:09,666 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer input -> K[dee],V[0 null hadoop hello] output -> K[dee],V[0 null hadoop hello] output -> K[hadoop],V[1 null:dee] output -> K[hello],V[1 null:dee] input -> K[hadoop],V[1 dee hive hello] output -> K[hadoop],V[1 dee hive hello] output -> K[hive],V[2 dee:hadoop] output -> K[hello],V[2 dee:hadoop] input -> K[hello],V[1 dee dee hadoop hive joe] output -> K[hello],V[1 dee dee hadoop hive joe] output -> K[dee],V[2 dee:hello] output -> K[hadoop],V[2 dee:hello] output -> K[hive],V[2 dee:hello] output -> K[joe],V[2 dee:hello] input -> K[hive],V[2147483647 null hadoop hello joe] output -> K[hive],V[2147483647 null hadoop hello joe] input -> K[joe],V[2147483647 null hive hello] output -> K[joe],V[2147483647 null hive hello] 2016-12-12 16:37:09,675 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-12 16:37:09,675 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-12 16:37:09,675 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-12 16:37:09,675 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 289; bufvoid = 104857600 2016-12-12 16:37:09,676 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600 2016-12-12 16:37:09,691 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-12 16:37:09,699 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local447108750_0002_m_000000_0 is done. And is in the process of committing 2016-12-12 16:37:09,704 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/shortestpath/1/part-r-00000:0+149 2016-12-12 16:37:09,705 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local447108750_0002_m_000000_0' done. 2016-12-12 16:37:09,705 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local447108750_0002_m_000000_0 2016-12-12 16:37:09,705 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-12 16:37:09,707 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-12 16:37:09,708 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local447108750_0002_r_000000_0 2016-12-12 16:37:09,714 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 16:37:09,856 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3f539d4b 2016-12-12 16:37:09,857 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@a7bc768 2016-12-12 16:37:09,862 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-12 16:37:09,865 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local447108750_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-12 16:37:09,871 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#2 about to shuffle output of map attempt_local447108750_0002_m_000000_0 decomp: 317 len: 321 to MEMORY 2016-12-12 16:37:09,874 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 317 bytes from map-output for attempt_local447108750_0002_m_000000_0 2016-12-12 16:37:09,876 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 317, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->317 2016-12-12 16:37:09,877 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-12 16:37:09,879 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 16:37:09,879 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-12 16:37:09,892 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 16:37:09,893 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 311 bytes 2016-12-12 16:37:09,896 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 317 bytes to disk to satisfy reduce memory limit 2016-12-12 16:37:09,898 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 321 bytes from disk 2016-12-12 16:37:09,898 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-12 16:37:09,898 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 16:37:09,901 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 311 bytes 2016-12-12 16:37:09,902 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. input -> K[dee] input -> V[2 dee:hello] input -> V[0 null hadoop hello] output -> K[dee],V[0 null hadoop hello] input -> K[hadoop] input -> V[1 null:dee] input -> V[1 dee hive hello] input -> V[2 dee:hello] output -> K[hadoop],V[1 null:dee hive hello] input -> K[hello] input -> V[1 dee dee hadoop hive joe] input -> V[2 dee:hadoop] input -> V[1 null:dee] output -> K[hello],V[1 dee dee hadoop hive joe] input -> K[hive] input -> V[2 dee:hadoop] input -> V[2 dee:hello] input -> V[2147483647 null hadoop hello joe] output -> K[hive],V[2 dee:hadoop hadoop hello joe] input -> K[joe] input -> V[2 dee:hello] input -> V[2147483647 null hive hello] output -> K[joe],V[2 dee:hello hive hello] 2016-12-12 16:37:09,929 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local447108750_0002_r_000000_0 is done. And is in the process of committing 2016-12-12 16:37:09,934 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 16:37:09,934 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local447108750_0002_r_000000_0 is allowed to commit now 2016-12-12 16:37:09,944 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local447108750_0002_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/shortestpath/2/_temporary/0/task_local447108750_0002_r_000000 2016-12-12 16:37:09,947 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-12 16:37:09,948 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local447108750_0002_r_000000_0' done. 2016-12-12 16:37:09,948 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local447108750_0002_r_000000_0 2016-12-12 16:37:09,948 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-12 16:37:10,526 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local447108750_0002 running in uber mode : false 2016-12-12 16:37:10,526 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-12 16:37:10,527 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local447108750_0002 completed successfully 2016-12-12 16:37:10,542 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 35 File System Counters FILE: Number of bytes read=3162 FILE: Number of bytes written=776144 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=5 Map output records=13 Map output bytes=289 Map output materialized bytes=321 Input split bytes=140 Combine input records=0 Combine output records=0 Reduce input groups=5 Reduce shuffle bytes=321 Reduce input records=13 Reduce output records=5 Spilled Records=26 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=677380096 PATH dee:hello=1 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=169 File Output Format Counters Bytes Written=159 zhouls.bigdata.myMapReduce.shortestpath.Reduce$PathCounter TARGET_NODE_DISTANCE_COMPUTED=2 ========================================== = Shortest path found, details as follows. = = Start node: dee = End node: joe = Hops: 2 = Path: dee:hello ========================================== 代码 package zhouls.bigdata.myMapReduce.shortestpath; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; import java.io.IOException; public class Map extends Mapper<Text, Text, Text, Text> { private Text outKey = new Text(); private Text outValue = new Text(); @Override protected void map(Text key, Text value, Context context) throws IOException, InterruptedException { Node node = Node.fromMR(value.toString()); System.out.println("input -> K[" + key + "],V[" + node + "]"); // output this node's key/value pair again to preserve the information // System.out.println( " output -> K[" + key + "],V[" + value + "]"); context.write(key, value); // only output the neighbor details if we have an actual distance // from the source node // if (node.isDistanceSet()) { // our neighbors are just a hop away // // create the backpointer, which will append our own // node name to the list // String backpointer = node.constructBackpointer(key.toString()); // go through all the nodes and propagate the distance to them // for (int i = 0; i < node.getAdjacentNodeNames().length; i++) { String neighbor = node.getAdjacentNodeNames()[i]; int neighborDistance = node.getDistance() + 1; // output the neighbor with the propagated distance and backpointer // outKey.set(neighbor); Node adjacentNode = new Node() .setDistance(neighborDistance) .setBackpointer(backpointer); outValue.set(adjacentNode.toString()); System.out.println( " output -> K[" + outKey + "],V[" + outValue + "]"); context.write(outKey, outValue); } } } } package zhouls.bigdata.myMapReduce.shortestpath; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.*; import java.io.IOException; public class Reduce extends Reducer<Text, Text, Text, Text> { public static enum PathCounter { TARGET_NODE_DISTANCE_COMPUTED, PATH } private Text outValue = new Text(); private String targetNode; protected void setup(Context context ) throws IOException, InterruptedException { targetNode = context.getConfiguration().get( Main.TARGET_NODE); } public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException { int minDistance = Node.INFINITE; System.out.println("input -> K[" + key + "]"); Node shortestAdjacentNode = null; Node originalNode = null; for (Text textValue : values) { System.out.println(" input -> V[" + textValue + "]"); Node node = Node.fromMR(textValue.toString()); if(node.containsAdjacentNodes()) { // the original data // originalNode = node; } if(node.getDistance() < minDistance) { minDistance = node.getDistance(); shortestAdjacentNode = node; } } if(shortestAdjacentNode != null) { originalNode.setDistance(minDistance); originalNode.setBackpointer(shortestAdjacentNode.getBackpointer()); } outValue.set(originalNode.toString()); System.out.println( " output -> K[" + key + "],V[" + outValue + "]"); context.write(key, outValue); if (minDistance != Node.INFINITE && targetNode.equals(key.toString())) { Counter counter = context.getCounter( PathCounter.TARGET_NODE_DISTANCE_COMPUTED); counter.increment(minDistance); context.getCounter(PathCounter.PATH.toString(), shortestAdjacentNode.getBackpointer()).increment(1); } } } package zhouls.bigdata.myMapReduce.shortestpath; import org.apache.commons.lang.StringUtils; import java.io.IOException; import java.util.Arrays; public class Node { private int distance = INFINITE; private String backpointer; private String[] adjacentNodeNames; public static int INFINITE = Integer.MAX_VALUE; public static final char fieldSeparator = '\t'; public int getDistance() { return distance; } public Node setDistance(int distance) { this.distance = distance; return this; } public String getBackpointer() { return backpointer; } public Node setBackpointer(String backpointer) { this.backpointer = backpointer; return this; } public String constructBackpointer(String name) { StringBuilder backpointers = new StringBuilder(); if (StringUtils.trimToNull(getBackpointer()) != null) { backpointers.append(getBackpointer()).append(":"); } backpointers.append(name); return backpointers.toString(); } public String[] getAdjacentNodeNames() { return adjacentNodeNames; } public Node setAdjacentNodeNames(String[] adjacentNodeNames) { this.adjacentNodeNames = adjacentNodeNames; return this; } public boolean containsAdjacentNodes() { return adjacentNodeNames != null; } public boolean isDistanceSet() { return distance != INFINITE; } @Override public String toString() { StringBuilder sb = new StringBuilder(); sb.append(distance) .append(fieldSeparator) .append(backpointer); if (getAdjacentNodeNames() != null) { sb.append(fieldSeparator) .append(StringUtils .join(getAdjacentNodeNames(), fieldSeparator)); } return sb.toString(); } public static Node fromMR(String value) throws IOException { String[] parts = StringUtils.splitPreserveAllTokens( value, fieldSeparator); if (parts.length < 2) { throw new IOException( "Expected 2 or more parts but received " + parts.length); } Node node = new Node() .setDistance(Integer.valueOf(parts[0])) .setBackpointer(StringUtils.trimToNull(parts[1])); if (parts.length > 2) { node.setAdjacentNodeNames(Arrays.copyOfRange(parts, 2, parts.length)); } return node; } } package zhouls.bigdata.myMapReduce.shortestpath; import org.apache.commons.io.*; import org.apache.commons.lang.*; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.*; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.input.*; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import java.io.*; import java.util.Iterator; public final class Main { public static final String TARGET_NODE = "shortestpath.targetnode"; public static void main(String... args) throws Exception { String startNode = "dee"; String targetNode = "joe"; // String inputFile = "hdfs://HadoopMaster:9000/shortestpath/shortestpath.txt"; // String outputDir = "hdfs://HadoopMaster:9000/out/shortestpath"; String inputFile = "./data/shortestpath/shortestpath.txt"; String outputDir = "./out/shortestpath"; iterate(startNode, targetNode, inputFile, outputDir); } public static Configuration conf = new Configuration(); static{ // conf.set("fs.defaultFS", "hdfs://HadoopMaster:9000"); // conf.set("yarn.resourcemanager.hostname", "HadoopMaster"); } public static void iterate(String startNode, String targetNode, String input, String output) throws Exception { Path outputPath = new Path(output); outputPath.getFileSystem(conf).delete(outputPath, true); outputPath.getFileSystem(conf).mkdirs(outputPath); Path inputPath = new Path(outputPath, "input.txt"); createInputFile(new Path(input), inputPath, startNode); int iter = 1; while (true) { Path jobOutputPath = new Path(outputPath, String.valueOf(iter)); System.out.println("======================================"); System.out.println("= Iteration: " + iter); System.out.println("= Input path: " + inputPath); System.out.println("= Output path: " + jobOutputPath); System.out.println("======================================"); if(findShortestPath(inputPath, jobOutputPath, startNode, targetNode)) { break; } inputPath = jobOutputPath; iter++; } } public static void createInputFile(Path file, Path targetFile, String startNode) throws IOException { FileSystem fs = file.getFileSystem(conf); OutputStream os = fs.create(targetFile); LineIterator iter = org.apache.commons.io.IOUtils .lineIterator(fs.open(file), "UTF8"); while (iter.hasNext()) { String line = iter.nextLine(); String[] parts = StringUtils.split(line); int distance = Node.INFINITE; if (startNode.equals(parts[0])) { distance = 0; } IOUtils.write(parts[0] + '\t' + String.valueOf(distance) + "\t\t", os); IOUtils.write(StringUtils.join(parts, '\t', 1, parts.length), os); IOUtils.write("\n", os); } os.close(); } public static boolean findShortestPath(Path inputPath, Path outputPath, String startNode, String targetNode) throws Exception { conf.set(TARGET_NODE, targetNode); Job job = new Job(conf); job.setJarByClass(Main.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(KeyValueTextInputFormat.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); FileInputFormat.setInputPaths(job, inputPath); FileOutputFormat.setOutputPath(job, outputPath); if (!job.waitForCompletion(true)) { throw new Exception("Job failed"); } Counter counter = job.getCounters() .findCounter(Reduce.PathCounter.TARGET_NODE_DISTANCE_COMPUTED); if(counter != null && counter.getValue() > 0) { CounterGroup group = job.getCounters().getGroup(Reduce.PathCounter.PATH.toString()); Iterator<Counter> iter = group.iterator(); iter.hasNext(); String path = iter.next().getName(); System.out.println("=========================================="); System.out.println("= Shortest path found, details as follows."); System.out.println("= "); System.out.println("= Start node: " + startNode); System.out.println("= End node: " + targetNode); System.out.println("= Hops: " + counter.getValue()); System.out.println("= Path: " + path); System.out.println("=========================================="); return true; } return false; } // public static String getNeighbor(String str){ // return str.split(",")[0]; // } // public static int getNeighborDis(String str){ // return Integer.parseInt(str.split(",")[1]); // } } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6165087.html，如需转载请自行联系原作者
2017-11-13
优秀的个人博客,低调大师

Hadoop MapReduce编程 API入门系列之分区和合并（十四）

代码 1 package zhouls.bigdata.myMapReduce.Star; 2 3 4 import java.io.IOException; 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.conf.Configured; 7 import org.apache.hadoop.fs.FileSystem; 8 import org.apache.hadoop.fs.Path; 9 import org.apache.hadoop.io.Text; 10 import org.apache.hadoop.mapreduce.Job; 11 import org.apache.hadoop.mapreduce.Mapper; 12 import org.apache.hadoop.mapreduce.Partitioner; 13 import org.apache.hadoop.mapreduce.Reducer; 14 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 15 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 16 import org.apache.hadoop.util.Tool; 17 import org.apache.hadoop.util.ToolRunner; 18 /** 19 * 20 * @function 统计分别统计出男女明星最大搜索指数 21 * @author 小讲 22 */ 23 24 /* 25 姓名 性别 搜索指数 26 李易峰 male 32670 27 朴信惠 female 13309 28 林心如 female 5242 29 黄海波 male 5505 30 成龙 male 7757 31 刘亦菲 female 14830 32 angelababy female 55083 33 王宝强 male 9472 34 郑爽 female 9279 35 周杰伦 male 42020 36 莫小棋 female 13978 37 朱一龙 male 10524 38 宋智孝 female 12494 39 吴京 male 6684 40 赵丽颖 female 24174 41 尹恩惠 female 5985 42 李金铭 female 5925 43 关之琳 female 7668 44 邓超 male 11532 45 钟汉良 male 8289 46 周润发 male 4808 47 甄子丹 male 5479 48 林妙可 female 5306 49 柳岩 female 8221 50 蔡琳 female 7320 51 张佳宁 female 6628 52 裴涩琪 female 5658 53 李晨 male 9559 54 周星驰 male 11483 55 杨紫 female 11094 56 全智贤 female 5336 57 张柏芝 female 9337 58 孙俪 female 7295 59 鲍蕾 female 5375 60 杨幂 female 20238 61 刘德华 male 19786 62 柯震东 male 6398 63 张国荣 male 5013 64 王阳 male 5169 65 李小龙 male 6859 66 林志颖 male 4512 67 林正英 male 5832 68 吴秀波 male 5668 69 陈伟霆 male 12817 70 陈奕迅 male 10472 71 赵又廷 male 5190 72 张馨予 female 35062 73 陈晓 male 17901 74 赵韩樱子 female 7077 75 乔振宇 male 8877 76 宋慧乔 female 5708 77 韩艺瑟 female 5426 78 张翰 male 7012 79 谢霆锋 male 6654 80 刘晓庆 female 5553 81 陈翔 male 7999 82 陈学冬 male 8829 83 秋瓷炫 female 6504 84 王祖蓝 male 6662 85 吴亦凡 male 16472 86 陈妍希 female 32590 87 倪妮 female 9278 88 高梓淇 male 7101 89 赵奕欢 female 7197 90 赵本山 male 12655 91 高圆圆 female 13688 92 陈赫 male 6820 93 鹿晗 male 32492 94 贾玲 female 5304 95 宋佳 female 6202 96 郭碧婷 female 5295 97 唐嫣 female 12055 98 杨蓉 female 10512 99 李钟硕 male 26278 100 郑秀晶 female 10479 101 熊黛林 female 26732 102 金秀贤 male 11370 103 古天乐 male 4954 104 黄晓明 male 10964 105 李敏镐 male 10512 106 王丽坤 female 5501 107 谢依霖 female 7000 108 陈冠希 male 9135 109 范冰冰 female 13734 110 姚笛 female 6953 111 彭于晏 male 14136 112 张学友 male 4578 113 谢娜 female 6886 114 胡歌 male 8015 115 古力娜扎 female 8858 116 黄渤 male 7825 117 周韦彤 female 7677 118 刘诗诗 female 16548 119 郭德纲 male 10307 120 郑恺 male 21145 121 赵薇 female 5339 122 李连杰 male 4621 123 宋茜 female 11164 124 任重 male 8383 125 李若彤 female 9968 126 127 128 得到： 129 angelababy female 55083 130 周杰伦 male 42020 131 */ 132 public class Star extends Configured implements Tool{ 133 /** 134 * @function Mapper 解析明星数据 135 * @input key=偏移量 value=明星数据 136 * @output key=gender value=name+hotIndex 137 */ 138 public static class ActorMapper extends Mapper<Object,Text,Text,Text>{ 139 //在这个例子里，第一个参数Object是Hadoop根据默认值生成的，一般是文件块里的一行文字的行偏移数，这些偏移数不重要，在处理时候一般用不上 140 public void map(Object key,Text value,Context context) throws IOException,InterruptedException{ 141 //拿：周杰伦 male 42020 142 //value=name+gender+hotIndex 143 String[] tokens = value.toString().split("\t");//使用分隔符\t，将数据解析为数组 tokens 144 String gender = tokens[1].trim();//性别,trim()是去除两边空格的方法 145 //tokens[0] tokens[1] tokens[2] 146 //周杰伦 male 42020 147 String nameHotIndex = tokens[0] + "\t" + tokens[2];//名称和关注指数 148 //输出key=gender value=name+hotIndex 149 context.write(new Text(gender), new Text(nameHotIndex));//写入gender是k2,nameHotIndex是v2 150 // context.write(gender,nameHotIndex);等价 151 //将gender和nameHotIndex写入到context中 152 } 153 } 154 155 156 157 /** 158 * @function Partitioner 根据sex选择分区 159 */ 160 public static class ActorPartitioner extends Partitioner<Text, Text>{ 161 @Override 162 public int getPartition(Text key, Text value, int numReduceTasks){ 163 String sex = key.toString();//按性别分区 164 165 // 默认指定分区 0 166 if(numReduceTasks==0) 167 return 0; 168 169 //性别为male 选择分区0 170 if(sex.equals("male")) 171 return 0; 172 //性别为female 选择分区1 173 if(sex.equals("female")) 174 return 1 % numReduceTasks; 175 //其他性别 选择分区2 176 else 177 return 2 % numReduceTasks; 178 179 } 180 } 181 182 183 184 /** 185 * @function 定义Combiner 合并 Mapper 输出结果 186 */ 187 public static class ActorCombiner extends Reducer<Text, Text, Text, Text>{ 188 private Text text = new Text(); 189 @Override 190 public void reduce(Text key, Iterable<Text> values, Context context)throws IOException, InterruptedException{ 191 int maxHotIndex = Integer.MIN_VALUE; 192 int hotIndex = 0; 193 String name=""; 194 for (Text val : values){//星型for循环，即把values的值传给Text val 195 String[] valTokens = val.toString().split("\\t"); 196 hotIndex = Integer.parseInt(valTokens[1]); 197 if(hotIndex>maxHotIndex){ 198 name = valTokens[0]; 199 maxHotIndex = hotIndex; 200 } 201 } 202 text.set(name+"\t"+maxHotIndex); 203 context.write(key, text); 204 } 205 } 206 207 208 209 /** 210 * @function Reducer 统计男、女明星最高搜索指数 211 * @input key=gender value=name+hotIndex 212 * @output key=name value=gender+hotIndex(max) 213 */ 214 public static class ActorReducer extends Reducer<Text,Text,Text,Text>{ 215 @Override 216 public void reduce(Text key, Iterable<Text> values, Context context)throws IOException, InterruptedException{ 217 int maxHotIndex = Integer.MIN_VALUE; 218 219 String name = " "; 220 int hotIndex = 0; 221 // 根据key，迭代 values 集合，求出最高搜索指数 222 for (Text val : values){//星型for循环，即把values的值传给Text val 223 String[] valTokens = val.toString().split("\\t"); 224 hotIndex = Integer.parseInt(valTokens[1]); 225 if (hotIndex > maxHotIndex){ 226 name = valTokens[0]; 227 maxHotIndex = hotIndex; 228 } 229 } 230 context.write(new Text(name), new Text(key + "\t"+ maxHotIndex));//写入name是k3,key + "\t"+ maxHotIndex是v3 231 // context.write(name,key + "\t"+ maxHotIndex);//等价 232 } 233 } 234 235 /** 236 * @function 任务驱动方法 237 * @param args 238 * @return 239 * @throws Exception 240 */ 241 242 public int run(String[] args) throws Exception{ 243 // TODO Auto-generated method stub 244 245 Configuration conf = new Configuration();//读取配置文件，比如core-site.xml等等 246 Path mypath = new Path(args[1]);//Path对象mypath 247 FileSystem hdfs = mypath.getFileSystem(conf);//FileSystem对象hdfs 248 if (hdfs.isDirectory(mypath)){ 249 hdfs.delete(mypath, true); 250 } 251 252 Job job = new Job(conf, "star");//新建一个任务 253 job.setJarByClass(Star.class);//主类 254 255 job.setNumReduceTasks(2);//reduce的个数设置为2 256 job.setPartitionerClass(ActorPartitioner.class);//设置Partitioner类 257 258 job.setMapperClass(ActorMapper.class);//Mapper 259 job.setMapOutputKeyClass(Text.class);//map 输出key类型 260 job.setMapOutputValueClass(Text.class);//map 输出value类型 261 262 job.setCombinerClass(ActorCombiner.class);//设置Combiner类 263 264 job.setReducerClass(ActorReducer.class);//Reducer 265 job.setOutputKeyClass(Text.class);//输出结果 key类型 266 job.setOutputValueClass(Text.class);//输出结果 value类型 267 268 FileInputFormat.addInputPath(job, new Path(args[0]));// 输入路径 269 FileOutputFormat.setOutputPath(job, new Path(args[1]));// 输出路径 270 job.waitForCompletion(true);//提交任务 271 return 0; 272 } 273 274 275 /** 276 * @function main 方法 277 * @param args 278 * @throws Exception 279 */ 280 public static void main(String[] args) throws Exception{ 281 // String[] args0 = { "hdfs://HadoopMaster:9000/star/star.txt", 282 // "hdfs://HadoopMaster:9000/out/star/" }; 283 String[] args0 = { "./data/star/star.txt", 284 "./out/star" }; 285 286 int ec = ToolRunner.run(new Configuration(), new Star(), args0); 287 System.exit(ec); 288 } 289 } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6165047.html，如需转载请自行联系原作者
2017-11-12
优秀的个人博客,低调大师

Hadoop HDFS编程 API入门系列之HDFS_HA（五）

代码 1 package zhouls.bigdata.myWholeHadoop.HDFS.hdfs3; 2 3 import java.io.FileInputStream; 4 import java.io.InputStream; 5 import java.io.OutputStream; 6 import java.net.URI; 7 8 import org.apache.hadoop.conf.Configuration; 9 import org.apache.hadoop.fs.FileSystem; 10 import org.apache.hadoop.fs.Path; 11 import org.apache.hadoop.io.IOUtils; 12 13 public class HDFS_HA { 14 15 16 public static void main(String[] args) throws Exception { 17 Configuration conf = new Configuration(); 18 conf.set("fs.defaultFS", "hdfs://ns1"); 19 conf.set("dfs.nameservices", "ns1"); 20 conf.set("dfs.ha.namenodes.ns1", "nn1,nn2"); 21 conf.set("dfs.namenode.rpc-address.ns1.nn1", "hadoop01:9000"); 22 conf.set("dfs.namenode.rpc-address.ns1.nn2", "hadoop02:9000"); 23 //conf.setBoolean(name, value); 24 conf.set("dfs.client.failover.proxy.provider.ns1", "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"); 25 FileSystem fs = FileSystem.get(new URI("hdfs://ns1"), conf, "hadoop"); 26 InputStream in =new FileInputStream("D://eclipse.rar"); 27 OutputStream out = fs.create(new Path("/eclipse")); 28 IOUtils.copyBytes(in, out, 4096, true); 29 } 30 } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6175601.html，如需转载请自行联系原作者
2017-11-12
优秀的个人博客,低调大师

Hadoop MapReduce编程 API入门系列之wordcount版本1（五）

这个很简单哈，编程的版本很多种。 代码版本1 1 package zhouls.bigdata.myMapReduce.wordcount5; 2 3 import java.io.IOException; 4 import java.util.StringTokenizer; 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.fs.Path; 7 import org.apache.hadoop.io.IntWritable; 8 import org.apache.hadoop.io.Text; 9 import org.apache.hadoop.mapreduce.Job; 10 import org.apache.hadoop.mapreduce.Mapper; 11 import org.apache.hadoop.mapreduce.Reducer; 12 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 13 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 14 15 public class WordCount 16 { 17 public static class TokenizerMapper 18 extends Mapper<Object, Text, Text, IntWritable>{ 19 20 private final static IntWritable one = new IntWritable(1); 21 private Text word = new Text(); 22 23 public void map(Object key, Text value, Context context 24 ) throws IOException, InterruptedException { 25 StringTokenizer itr = new StringTokenizer(value.toString()); 26 while (itr.hasMoreTokens()) { 27 word.set(itr.nextToken()); 28 context.write(word, one); 29 } 30 } 31 } 32 33 public static class IntSumReducer 34 extends Reducer<Text,IntWritable,Text,IntWritable> { 35 private IntWritable result = new IntWritable(); 36 37 public void reduce(Text key, Iterable<IntWritable> values, 38 Context context 39 ) throws IOException, InterruptedException { 40 int sum = 0; 41 for (IntWritable val : values) { 42 sum += val.get(); 43 } 44 result.set(sum); 45 context.write(key, result); 46 } 47 } 48 49 public static void main(String[] args) throws Exception { 50 Configuration conf = new Configuration(); 51 Job job = Job.getInstance(conf, "word count"); 52 job.setJarByClass(WordCount.class); 53 job.setMapperClass(TokenizerMapper.class); 54 job.setCombinerClass(IntSumReducer.class); 55 job.setReducerClass(IntSumReducer.class); 56 job.setOutputKeyClass(Text.class); 57 job.setOutputValueClass(IntWritable.class); 58 // FileInputFormat.addInputPath(job, new Path("hdfs:/HadoopMaster:9000/wc.txt")); 59 // FileOutputFormat.setOutputPath(job, new Path("hdfs:/HadoopMaster:9000/out/wordcount")); 60 FileInputFormat.addInputPath(job, new Path("./data/wc.txt")); 61 FileOutputFormat.setOutputPath(job, new Path("./out/WordCount")); 62 System.exit(job.waitForCompletion(true) ? 0 : 1); 63 } 64 } 代码版本3 1 package com.dajiangtai.Hadoop.MapReduce; 2 3 4 import java.io.IOException; 5 import java.util.StringTokenizer; 6 7 import org.apache.hadoop.conf.Configuration; 8 import org.apache.hadoop.fs.FileSystem; 9 import org.apache.hadoop.fs.Path; 10 import org.apache.hadoop.io.IntWritable; 11 import org.apache.hadoop.io.Text; 12 import org.apache.hadoop.mapreduce.Job; 13 import org.apache.hadoop.mapreduce.Mapper; 14 import org.apache.hadoop.mapreduce.Reducer; 15 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 16 import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; 17 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 18 import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; 19 20 21 @SuppressWarnings("unused") 22 public class WordCount {//2017最新详解版 23 24 public static class TokenizerMapper extends 25 Mapper<Object, Text, Text, IntWritable> 26 // 为什么这里k1要用Object、Text、IntWritable等，而不是java的string啊、int啊类型，当然，你可以用其他的，这样用的好处是，因为它里面实现了序列化和反序列化。 27 // 可以让在节点间传输和通信效率更高。这就为什么hadoop本身的机制类型的诞生。 28 29 30 //这个Mapper类是一个泛型类型，它有四个形参类型，分别指定map函数的输入键、输入值、输出键、输出值的类型。hadoop没有直接使用Java内嵌的类型，而是自己开发了一套可以优化网络序列化传输的基本类型。这些类型都在org.apache.hadoop.io包中。 31 //比如这个例子中的Object类型，适用于字段需要使用多种类型的时候，Text类型相当于Java中的String类型，IntWritable类型相当于Java中的Integer类型 32 { 33 //定义两个变量或者说是定义两个对象，叫法都可以 34 private final static IntWritable one = new IntWritable(1);//这个1表示每个单词出现一次，map的输出value就是1. 35 //因为，v1是单词出现次数，直接对one赋值为1 36 private Text word = new Text(); 37 38 public void map(Object key, Text value, Context context) 39 //context它是mapper的一个内部类，简单的说顶级接口是为了在map或是reduce任务中跟踪task的状态，很自然的MapContext就是记录了map执行的上下文，在mapper类中，这个context可以存储一些job conf的信息，比如job运行时参数等，我们可以在map函数中处理这个信息，这也是Hadoop中参数传递中一个很经典的例子，同时context作为了map和reduce执行中各个函数的一个桥梁，这个设计和Java web中的session对象、application对象很相似 40 //简单的说context对象保存了作业运行的上下文信息，比如：作业配置信息、InputSplit信息、任务ID等 41 //我们这里最直观的就是主要用到context的write方法。 42 //说白了，context起到的是连接map和reduce的桥梁。起到上下文的作用！ 43 44 throws IOException, InterruptedException { 45 //The tokenizer uses the default delimiter set, which is " \t\n\r": the space character, the tab character, the newline character, the carriage-return character 46 StringTokenizer itr = new StringTokenizer(value.toString());//将Text类型的value转化成字符串类型 47 //StringTokenizer是字符串分隔解析类型,StringTokenizer 用来分割字符串，你可以指定分隔符，比如','，或者空格之类的字符。 48 49 50 //使用StringTokenizer类将字符串“hello,java,delphi,asp,PHP”分解为三个单词 51 // 程序的运行结果为: 52 // hello 53 // java 54 // delphi 55 // asp 56 // 57 // php 58 59 60 while (itr.hasMoreTokens()) {//hasMoreTokens() 方法是用来测试是否有此标记生成器的字符串可用更多的标记。 61 // 实际上就是java.util.StringTokenizer.hasMoreTokens() 62 // hasMoreTokens() 方法是用来测试是否有此标记生成器的字符串可用更多的标记。 63 //java.util.StringTokenizer.hasMoreTokens() 64 65 66 word.set(itr.nextToken());//nextToken()这是 StringTokenizer 类下的一个方法,nextToken() 用于返回下一个匹配的字段。 67 context.write(word, one); 68 } 69 } 70 } 71 72 73 74 75 public static class IntSumReducer extends 76 Reducer<Text, IntWritable, Text, IntWritable> { 77 private IntWritable result = new IntWritable(); 78 public void reduce(Text key, Iterable<IntWritable> values, 79 Context context) throws IOException, InterruptedException { 80 //我们这里最直观的就是主要用到context的write方法。 81 //说白了，context起到的是连接map和reduce的桥梁。起到上下文的作用！ 82 83 int sum = 0; 84 for (IntWritable val : values) {//叫做增强的for循环，也叫for星型循环 85 sum += val.get(); 86 } 87 result.set(sum); 88 context.write(key, result); 89 } 90 } 91 92 public static void main(String[] args) throws Exception { 93 Configuration conf = new Configuration();//程序里，只需写这么一句话，就会加载到hadoop的配置文件了 94 //Configuration类代表作业的配置，该类会加载mapred-site.xml、hdfs-site.xml、core-site.xml等配置文件。 95 //删除已经存在的输出目录 96 Path mypath = new Path("hdfs://djt002:9000/outData/wordcount");//输出路径 97 FileSystem hdfs = mypath.getFileSystem(conf);//程序里，只需写这么一句话，就可以获取到文件系统了。 98 //FileSystem里面包括很多系统，不局限于hdfs，是因为，程序读到conf，哦，原来是hadoop集群啊。这时，才认知到是hdfs 99 100 //如果文件系统中存在这个输出路径，则删除掉，保证输出目录不能提前存在。 101 if (hdfs.isDirectory(mypath)) { 102 hdfs.delete(mypath, true); 103 } 104 105 //job对象指定了作业执行规范，可以用它来控制整个作业的运行。 106 Job job = Job.getInstance();// new Job(conf, "word count"); 107 job.setJarByClass(WordCount.class);//我们在hadoop集群上运行作业的时候，要把代码打包成一个jar文件，然后把这个文件 108 //传到集群上，然后通过命令来执行这个作业，但是命令中不必指定JAR文件的名称，在这条命令中通过job对象的setJarByClass（） 109 //中传递一个主类就行，hadoop会通过这个主类来查找包含它的JAR文件。 110 111 job.setMapperClass(TokenizerMapper.class); 112 //job.setReducerClass(IntSumReducer.class); 113 job.setCombinerClass(IntSumReducer.class);//Combiner最终不能影响reduce输出的结果 114 // 这句话要好好理解!!! 115 116 117 118 job.setOutputKeyClass(Text.class); 119 job.setOutputValueClass(IntWritable.class); 120 //一般情况下mapper和reducer的输出的数据类型是一样的，所以我们用上面两条命令就行，如果不一样，我们就可以用下面两条命令单独指定mapper的输出key、value的数据类型 121 //job.setMapOutputKeyClass(Text.class); 122 //job.setMapOutputValueClass(IntWritable.class); 123 //hadoop默认的是TextInputFormat和TextOutputFormat,所以说我们这里可以不用配置。 124 //job.setInputFormatClass(TextInputFormat.class); 125 //job.setOutputFormatClass(TextOutputFormat.class); 126 127 FileInputFormat.addInputPath(job, new Path( 128 "hdfs://djt002:9000/inputData/wordcount/wc.txt"));//FileInputFormat.addInputPath（）指定的这个路径可以是单个文件、一个目录或符合特定文件模式的一系列文件。 129 //从方法名称可以看出，可以通过多次调用这个方法来实现多路径的输入。 130 FileOutputFormat.setOutputPath(job, new Path( 131 "hdfs://djt002:9000/outData/wordcount"));//只能有一个输出路径，该路径指定的就是reduce函数输出文件的写入目录。 132 //特别注意：输出目录不能提前存在，否则hadoop会报错并拒绝执行作业，这样做的目的是防止数据丢失，因为长时间运行的作业如果结果被意外覆盖掉，那肯定不是我们想要的 133 System.exit(job.waitForCompletion(true) ? 0 : 1); 134 //使用job.waitForCompletion（）提交作业并等待执行完成，该方法返回一个boolean值，表示执行成功或者失败，这个布尔值被转换成程序退出代码0或1，该布尔参数还是一个详细标识，所以作业会把进度写到控制台。 135 //waitForCompletion(）提交作业后，每秒会轮询作业的进度，如果发现和上次报告后有改变，就把进度报告到控制台，作业完成后，如果成功就显示作业计数器，如果失败则把导致作业失败的错误输出到控制台 136 } 137 } 138 139 //TextInputFormat是hadoop默认的输入格式，这个类继承自FileInputFormat,使用这种输入格式，每个文件都会单独作为Map的输入，每行数据都会生成一条记录，每条记录会表示成<key，value>的形式。 140 //key的值是每条数据记录在数据分片中的字节偏移量，数据类型是LongWritable. 141 //value的值为每行的内容，数据类型为Text。 142 // 143 //实际上InputFormat（）是用来生成可供Map处理的<key，value>的。 144 //InputSplit是hadoop中用来把输入数据传送给每个单独的Map(也就是我们常说的一个split对应一个Map), 145 //InputSplit存储的并非数据本身，而是一个分片长度和一个记录数据位置的数组。 146 //生成InputSplit的方法可以通过InputFormat（）来设置。 147 //当数据传给Map时，Map会将输入分片传送给InputFormat（），InputFormat()则调用getRecordReader()生成RecordReader,RecordReader则再通过creatKey()和creatValue()创建可供Map处理的<key，value>对。 148 // 149 //OutputFormat() 150 //默认的输出格式为TextOutputFormat。它和默认输入格式类似，会将每条记录以一行的形式存入文本文件。它的键和值可以是任意形式的，因为程序内部会调用toString()将键和值转化为String类型再输出。 代码版本2 1 package zhouls.bigdata.myMapReduce.wordcount5; 2 3 import java.io.IOException; 4 import java.util.StringTokenizer; 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.fs.FileSystem; 7 import org.apache.hadoop.fs.Path; 8 import org.apache.hadoop.io.IntWritable; 9 import org.apache.hadoop.io.Text; 10 import org.apache.hadoop.mapreduce.Job; 11 import org.apache.hadoop.mapreduce.Mapper; 12 import org.apache.hadoop.mapreduce.Reducer; 13 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 14 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 15 import org.apache.hadoop.util.Tool; 16 import org.apache.hadoop.util.ToolRunner; 17 18 19 20 public class WordCount implements Tool 21 { 22 public static class TokenizerMapper 23 extends Mapper<Object, Text, Text, IntWritable>{ 24 25 private final static IntWritable one = new IntWritable(1); 26 private Text word = new Text(); 27 28 public void map(Object key, Text value, Context context 29 ) throws IOException, InterruptedException { 30 StringTokenizer itr = new StringTokenizer(value.toString()); 31 while (itr.hasMoreTokens()) { 32 word.set(itr.nextToken()); 33 context.write(word, one); 34 } 35 } 36 } 37 38 public static class IntSumReducer 39 extends Reducer<Text,IntWritable,Text,IntWritable> { 40 private IntWritable result = new IntWritable(); 41 42 public void reduce(Text key, Iterable<IntWritable> values, 43 Context context 44 ) throws IOException, InterruptedException { 45 int sum = 0; 46 for (IntWritable val : values) { 47 sum += val.get(); 48 } 49 result.set(sum); 50 context.write(key, result); 51 } 52 } 53 54 55 public int run(String[] arg0) throws Exception { 56 Configuration conf = new Configuration(); 57 //2删除已经存在的输出目录 58 Path mypath = new Path(arg0[1]);//下标为1，即是输出路径 59 FileSystem hdfs = mypath.getFileSystem(conf);//获取文件系统 60 if (hdfs.isDirectory(mypath)) 61 {//如果文件系统中存在这个输出路径，则删除掉 62 hdfs.delete(mypath, true); 63 } 64 65 Job job = Job.getInstance(conf, "word count"); 66 job.setJarByClass(WordCount.class); 67 job.setMapperClass(TokenizerMapper.class); 68 job.setCombinerClass(IntSumReducer.class); 69 job.setReducerClass(IntSumReducer.class); 70 job.setOutputKeyClass(Text.class); 71 job.setOutputValueClass(IntWritable.class); 72 73 74 FileInputFormat.addInputPath(job, new Path(arg0[0]));// 文件输入路径 75 FileOutputFormat.setOutputPath(job, new Path(arg0[1]));// 文件输出路径 76 job.waitForCompletion(true); 77 78 return 0; 79 80 } 81 82 83 public static void main(String[] args) throws Exception { 84 85 //集群路径 86 // String[] args0 = { "hdfs:/HadoopMaster:9000/wc.txt", 87 // "hdfs:/HadoopMaster:9000/out/wordcount"}; 88 89 //本地路径 90 String[] args0 = { "./data/wc.txt", 91 "./out/WordCount"}; 92 int ec = ToolRunner.run( new Configuration(), new WordCount(), args0); 93 System. exit(ec); 94 } 95 96 97 @Override 98 public Configuration getConf() { 99 // TODO Auto-generated method stub 100 return null; 101 } 102 103 104 @Override 105 public void setConf(Configuration arg0) { 106 // TODO Auto-generated method stub 107 108 } 109 }

2017-11-12
优秀的个人博客,低调大师

HBase编程 API入门系列之modify（管理端而言）（10）

这里，我带领大家，学习更高级的，因为，在开发中，尽量不能去服务器上修改表。 所以，在管理端来修改HBase表。采用线程池的方式（也是生产开发里首推的） package zhouls.bigdata.HbaseProject.Pool; import java.io.IOException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.client.HConnection; import org.apache.hadoop.hbase.client.HConnectionManager; public class TableConnection { private TableConnection(){ } private static HConnection connection = null; public static HConnection getConnection(){ if(connection == null){ ExecutorService pool = Executors.newFixedThreadPool(10);//建立一个固定大小的线程池 Configuration conf = HBaseConfiguration.create(); conf.set("hbase.zookeeper.quorum","HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); try{ connection = HConnectionManager.createConnection(conf,pool);//创建连接时，拿到配置文件和线程池 }catch (IOException e){ } } return connection; } } 1、修改HBase表 暂时，有错误 package zhouls.bigdata.HbaseProject.Pool; import java.io.IOException; import zhouls.bigdata.HbaseProject.Pool.TableConnection; import javax.xml.transform.Result; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.Cell; import org.apache.hadoop.hbase.CellUtil; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.MasterNotRunningException; import org.apache.hadoop.hbase.NamespaceDescriptor; import org.apache.hadoop.hbase.TableName; import org.apache.hadoop.hbase.ZooKeeperConnectionException; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.HTableInterface; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.util.Bytes; public class HBaseTest { public static void main(String[] args) throws Exception { // HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_04 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy1"));//列簇是f，列修饰符是name，值是Andy0 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 // table.put(put); // table.close(); // Get get = new Get(Bytes.toBytes("row_04")); // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 // org.apache.hadoop.hbase.client.Result rest = table.get(get); // System.out.println(rest.toString()); // table.close(); // Delete delete = new Delete(Bytes.toBytes("row_2")); // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); // table.delete(delete); // table.close(); // Delete delete = new Delete(Bytes.toBytes("row_04")); //// delete.deleteColumn(Bytes.toBytes("f"), Bytes.toBytes("name"));//deleteColumn是删除某一个列簇里的最新时间戳版本。 // delete.deleteColumns(Bytes.toBytes("f"), Bytes.toBytes("name"));//delete.deleteColumns是删除某个列簇里的所有时间戳版本。 // table.delete(delete); // table.close(); // Scan scan = new Scan(); // scan.setStartRow(Bytes.toBytes("row_01"));//包含开始行键 // scan.setStopRow(Bytes.toBytes("row_03"));//不包含结束行键 // scan.addColumn(Bytes.toBytes("f"), Bytes.toBytes("name")); // ResultScanner rst = table.getScanner(scan);//整个循环 // System.out.println(rst.toString()); // for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ) // { // for(Cell cell:next.rawCells()){//某个row key下的循坏 // System.out.println(next.toString()); // System.out.println("family:" + Bytes.toString(CellUtil.cloneFamily(cell))); // System.out.println("col:" + Bytes.toString(CellUtil.cloneQualifier(cell))); // System.out.println("value" + Bytes.toString(CellUtil.cloneValue(cell))); // } // } // table.close(); HBaseTest hbasetest =new HBaseTest(); // hbasetest.insertValue(); // hbasetest.getValue(); // hbasetest.delete(); // hbasetest.scanValue(); hbasetest.createTable("test_table3", "f");//先判断表是否存在，再来创建HBase表（生产开发首推） // hbasetest.deleteTable("test_table4");//先判断表是否存在，再来删除HBase表（生产开发首推） // hbasetest.modifyTable("test_table","row_02","f",'f:age'); } //生产开发中，建议这样用线程池做 // public void insertValue() throws Exception{ // HTableInterface table = TableConnection.getConnection().getTable(TableName.valueOf("test_table")); // Put put = new Put(Bytes.toBytes("row_01"));//行键是row_01 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy0")); // table.put(put); // table.close(); // } //生产开发中，建议这样用线程池做 // public void getValue() throws Exception{ // HTableInterface table = TableConnection.getConnection().getTable(TableName.valueOf("test_table")); // Get get = new Get(Bytes.toBytes("row_03")); // get.addColumn(Bytes.toBytes("f"), Bytes.toBytes("name")); // org.apache.hadoop.hbase.client.Result rest = table.get(get); // System.out.println(rest.toString()); // table.close(); // } // //生产开发中，建议这样用线程池做 // public void delete() throws Exception{ // HTableInterface table = TableConnection.getConnection().getTable(TableName.valueOf("test_table")); // Delete delete = new Delete(Bytes.toBytes("row_01")); // delete.deleteColumn(Bytes.toBytes("f"), Bytes.toBytes("name"));//deleteColumn是删除某一个列簇里的最新时间戳版本。 //// delete.deleteColumns(Bytes.toBytes("f"), Bytes.toBytes("name"));//delete.deleteColumns是删除某个列簇里的所有时间戳版本。 // table.delete(delete); // table.close(); // // } //生产开发中，建议这样用线程池做 // public void scanValue() throws Exception{ // HTableInterface table = TableConnection.getConnection().getTable(TableName.valueOf("test_table")); // Scan scan = new Scan(); // scan.setStartRow(Bytes.toBytes("row_02"));//包含开始行键 // scan.setStopRow(Bytes.toBytes("row_04"));//不包含结束行键 // scan.addColumn(Bytes.toBytes("f"), Bytes.toBytes("name")); // ResultScanner rst = table.getScanner(scan);//整个循环 // System.out.println(rst.toString()); // for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ) // { // for(Cell cell:next.rawCells()){//某个row key下的循坏 // System.out.println(next.toString()); // System.out.println("family:" + Bytes.toString(CellUtil.cloneFamily(cell))); // System.out.println("col:" + Bytes.toString(CellUtil.cloneQualifier(cell))); // System.out.println("value" + Bytes.toString(CellUtil.cloneValue(cell))); // } // } // table.close(); // } // //生产开发中，建议这样用线程池做 public void createTable(String tableName,String family) throws MasterNotRunningException, ZooKeeperConnectionException, IOException{ Configuration conf = HBaseConfiguration.create(getConfig()); HBaseAdmin admin = new HBaseAdmin(conf); HTableDescriptor tableDesc = new HTableDescriptor(TableName.valueOf(tableName)); HColumnDescriptor hcd = new HColumnDescriptor(family); hcd.setMaxVersions(3); // hcd.set//很多的带创建操作，我这里只抛砖引玉的作用 tableDesc.addFamily(hcd); if (!admin.tableExists(tableName)){ admin.createTable(tableDesc); }else{ System.out.println(tableName + "exist"); } admin.close(); } public void modifyTable(String tableName,String rowkey,String family,HColumnDescriptor hColumnDescriptor) throws MasterNotRunningException, ZooKeeperConnectionException, IOException{ Configuration conf = HBaseConfiguration.create(getConfig()); HBaseAdmin admin = new HBaseAdmin(conf); // HTableDescriptor tableDesc = new HTableDescriptor(TableName.valueOf(tableName)); HColumnDescriptor hcd = new HColumnDescriptor(family); // NamespaceDescriptor nsd = admin.getNamespaceDescriptor(tableName); // nsd.setConfiguration("hbase.namespace.quota.maxregion", "10"); // nsd.setConfiguration("hbase.namespace.quota.maxtables", "10"); if (admin.tableExists(tableName)){ admin.modifyColumn(tableName, hcd); // admin.modifyTable(tableName, tableDesc); // admin.modifyNamespace(nsd); }else{ System.out.println(tableName + "not exist"); } admin.close(); } //生产开发中，建议这样用线程池做 // public void deleteTable(String tableName)throws MasterNotRunningException, ZooKeeperConnectionException, IOException{ // Configuration conf = HBaseConfiguration.create(getConfig()); // HBaseAdmin admin = new HBaseAdmin(conf); // if (admin.tableExists(tableName)){ // admin.disableTable(tableName); // admin.deleteTable(tableName); // }else{ // System.out.println(tableName + "not exist"); // } // admin.close(); // } public static Configuration getConfig(){ Configuration configuration = new Configuration(); // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); return configuration; } } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6159995.html，如需转载请自行联系原作者
2017-11-12
优秀的个人博客,低调大师

Hadoop MapReduce编程 API入门系列之wordcount版本3（七）

代码 1 package zhouls.bigdata.myMapReduce.wordcount3; 2 3 import java.io.IOException; 4 5 import org.apache.hadoop.io.IntWritable; 6 import org.apache.hadoop.io.LongWritable; 7 import org.apache.hadoop.io.Text; 8 import org.apache.hadoop.mapreduce.Mapper; 9 import org.apache.hadoop.util.StringUtils; 10 11 public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>{ 12 13 //该方法循环调用，从文件的split中读取每行调用一次，把该行所在的下标为key，该行的内容为value 14 protected void map(LongWritable key, Text value, 15 Context context) 16 throws IOException, InterruptedException { 17 String[] words = StringUtils.split(value.toString(), ' '); 18 for(String w :words){ 19 context.write(new Text(w), new IntWritable(1)); 20 } 21 } 22 } 1 package zhouls.bigdata.myMapReduce.wordcount3; 2 3 import java.io.IOException; 4 5 import org.apache.hadoop.io.IntWritable; 6 import org.apache.hadoop.io.Text; 7 import org.apache.hadoop.mapreduce.Reducer; 8 9 public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable>{ 10 11 //每组调用一次，这一组数据特点：key相同，value可能有多个。 12 protected void reduce(Text arg0, Iterable<IntWritable> arg1, 13 Context arg2) 14 throws IOException, InterruptedException { 15 int sum =0; 16 for(IntWritable i: arg1){ 17 sum=sum+i.get(); 18 } 19 arg2.write(arg0, new IntWritable(sum)); 20 } 21 } 1 package zhouls.bigdata.myMapReduce.wordcount3; 2 3 4 import org.apache.hadoop.conf.Configuration; 5 import org.apache.hadoop.fs.FileSystem; 6 import org.apache.hadoop.fs.Path; 7 import org.apache.hadoop.io.IntWritable; 8 import org.apache.hadoop.io.Text; 9 import org.apache.hadoop.mapreduce.Job; 10 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 11 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 12 13 public class RunJob { 14 15 public static void main(String[] args) { 16 Configuration config =new Configuration(); 17 18 try { 19 FileSystem fs =FileSystem.get(config); 20 21 Job job =Job.getInstance(config); 22 job.setJarByClass(RunJob.class); 23 24 job.setJobName("wc"); 25 26 job.setMapperClass(WordCountMapper.class); 27 job.setReducerClass(WordCountReducer.class); 28 29 job.setMapOutputKeyClass(Text.class); 30 job.setMapOutputValueClass(IntWritable.class); 31 32 FileInputFormat.addInputPath(job, new Path("./data/wc.txt")); 33 34 Path outpath =new Path("./out/WordCountout"); 35 if(fs.exists(outpath)){ 36 fs.delete(outpath, true); 37 } 38 FileOutputFormat.setOutputPath(job, outpath); 39 40 boolean f= job.waitForCompletion(true); 41 if(f){ 42 System.out.println("job任务执行成功"); 43 } 44 } catch (Exception e) { 45 e.printStackTrace(); 46 } 47 } 48 }
2017-11-11
优秀的个人博客,低调大师

openstack 调用API 实现云主机的IO 控制，CGroup 策略

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 # vim: tabstop= 4 shiftwidth= 4 softtabstop= 4 # Copyright (c) 2011 X.commerce, a business unit of eBay Inc. # Copyright 2010 United States Government as represented by the # Administrator of the National Aeronautics and Space Administration. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License" ); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http: //www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. # Single virtual machine io limit , reader 70 /MB , writer 50 /MB "" " curl -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}' -H "Content-type: application/json" http: //127.0.0.1:35357/v2.0/tokens | python -m json.tool curl -v -d '{"ioctlAction": {"instance_name":"instance-0000000c","limit_reader":"40MB","limit_writer":"60MB"}}' -i http: //127.0.0.1:8774/v2/ba5a6a86a8224a70ba0ffde747d8a724/servers/e6b9cae7-8732-405d-9aaf-575643b4bbef/action -X POST -H "X-Auth-Project-Id: ba5a6a86a8224a70ba0ffde747d8a724" -H "Accept: application/json" -H "X-Auth-Token: 6f0c37595f384fd4a18766825cf49013" -H "Content-Type: application/json" "" " import time,sys,os,libvirt, trace back,re,subprocess,json from nova import log as logging from nova import flags from nova.periodic.blkio import config from nova import utils from nova.periodic.blkio import utils as blkio_utils FLAGS = flags.FLAGS LOG = logging.getLogger(__name__) class ActionIoctl(object): def __init__(self,instance_name,limit_reader,limit_writer): self.conn = libvirt.open(None) self.limitReader = int (limit_reader[ 0 :- 2 ]) self.limitWriter = int (limit_writer[ 0 :- 2 ]) self.name = instance_name self.path = FLAGS.instances_path def diskDeviceNum(self,disk): "" " Func info: Returns the number of hard disk,other is 8:0 " "" if disk == '/dev/sda1' or disk == '/dev/hda1' or disk[:- 1 ] == '/dev/sda' or disk[:- 1 ] == '/dev/hda' : return '8:0' elif disk == '/dev/sdb1' or disk == '/dev/hdb1' or disk[:- 1 ] == '/dev/sdb' or disk[:- 1 ] == '/dev/hdb' : return '8:16' elif disk == '/dev/sdc1' or disk == '/dev/hdc1' or disk[:- 1 ] == '/dev/sdc' or disk[:- 1 ] == '/dev/hdc' : return '8:32' elif disk == '/dev/sdd1' or disk == '/dev/hdd1' or disk[:- 1 ] == '/dev/sdd' or disk[:- 1 ] == '/dev/hdd' : return '8:48' else : return '8:16' def vmsPIDall(self): "" " Getting all vm port, return to " "" pid = list() try : cmds = "" "ps -ef |grep uuid|grep -v grep |awk '{print $2}'" "" pid=os.popen(cmds).readlines() return pid except Exception,e: LOG.error(_( '[-william-] Error %s' %e)) return def cgroupPath(self): '' ' Func info : Return Cgroup path ,Because the system is not the same, the Cgroup path is different also. By default , have RedHat, Centos, Ubuntu linux ... osType : The current system version , And return to ... Cpath : Cgroup path '' ' osType = open( '/etc/issue' ).readlines() for i in osType: if re.match( 'Ubuntu' ,i): Cpath = '/sys/fs/cgroup/' return Cpath elif re.match( 'CentOS' ,i): Cpath = '/cgroup/' return Cpath elif re.match( 'Red Hat' ,i): Cpath = '/cgroup/' return Cpath else : return '/sys/fs/cgroup/' def exeCmd(self,cmds): '' ' Perform system command ' '' LOG.info(_( 'exeCmd %s' %cmds)) try : exec_process = subprocess.Popen(cmds, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True) exec_process.wait() rest = exec_process.stdout.read() err = exec_process.stderr.read() if err == '' : return rest else : LOG.error(_( '[-william-] return values is null' )) except Exception,e: LOG.error(_( '[-william-] Error %s' ) %e) return '' ' Geting vmName name and pid ' '' def get_vm_pid(self,name): pid = self.conn.listDomainsID() try : pid = os.popen( "" "ps -ef |grep %s |grep -v grep |awk '{print $2}'" "" %name).readlines() return int (pid[ 0 ]) except: LOG.error(_( '[-william-] Error %s' \ % trace back.format_exc())) return def _vm_pids(self): try : return os.popen( 'pidof kvm' ).read().split() except: return def work(self): "" " Func info : working func ................ Cpath : is _Cpath function return values ... path : The current virtual machine IMG file path vm_count: All of the current node number of virtual machines size : Limit size mountInfo : mounting point "" " Cpath = self.cgroupPath() vm_pid = self.get_vm_pid(self.name) LOG.info(_( '[- william -] %s , %s' %(self.name,vm_pid))) r_size = self.limitReader * 1000 * 1000 w_size = self.limitWriter * 1000 * 1000 device = blkio_utils.instance_path(self.path) diskDeviceNum = self.diskDeviceNum(device) vmTaskPath = "%s/blkio/%s" %(Cpath,self.name) utils.execute( "mkdir" , "-p" ,vmTaskPath,run_as_root=True) utils.execute( "tee" , "%s/blkio.throttle.read_bps_device" %vmTaskPath,\ process_input = "%s %s" %(diskDeviceNum,r_size),\ run_as_root=True) utils.execute( "tee" , "%s/blkio.throttle.write_bps_device" %vmTaskPath, \ process_input = "%s %s" %(diskDeviceNum,w_size) ,\ run_as_root=True) LOG.info(_( "" " [-william-] echo '%s, %s, %s , %s , %s ' write oK " "" %(self.path , device, diskDeviceNum, w_size, r_size))) utils.execute( 'tee' , '%s/tasks' %vmTaskPath,\ process_input= str(vm_pid) ,\ run_as_root=True) LOG.info(_( '[- william-] append pid %s in task' %vm_pid)) if __name__ == "__main__" : sc = ActionIoctl( 'instance-0000000c' , '40MB' , '30MB' ) sc.work() 本文转自 swq499809608 51CTO博客，原文链接:http://blog.51cto.com/swq499809608/1412022
2017-11-11

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

时间: 2026-04-11 大小: 211.28KB 下载: 56次
Spring

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

时间: 2025-12-17 大小: 5MB 下载: 1次
Rocky Linux

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

时间: 2026-04-12 大小: 1.42GB 下载: 4次
Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。

时间: 2025-12-03 大小: 9.84MB 下载: 39次
热门标签
DeepSeek Jdk25 OpenAi HarmonyOS6 Nacos3 SpringBoot4 SpringCloud Docker Kubernetes Service Mesh Redis Gemini3 Rocky

欢迎您来访！

登录后，您将获得更多功能！
热门文章
联系我们

有任何问题或合作意向欢迎联系我们

Email: 99873273@qq.com

QQ: 99873273

用户登录
用户注册