搜索[快速入门]结果-低调大师优秀个人博客

精选列表

搜索[快速入门]，共10000篇文章

LeadTools Android 入门教学——运行第一个Android Demo

LeadTools 有很多Windows平台下的Demo，非常全面，但是目前开发手机应用的趋势也越来越明显，LeadTools也给大家提供了10个Android的Demo，这篇文章将会教你如何运行第一个Android Demo，我想只有把Demo跑起来了，才能更好地使用LeadTools开发自己的原生Android应用，当然大家也可以把Demo改吧改吧就是自己的原生Android应用。说到Android平台，LeadTools提供了非常全面的支持，但是要将第一个Demo跑起来，License是不可少的。 LeadTools的Android平台，使用的是Runtime License，意思就是需要在程序运行开始时使用代码将得到的License文件读入并设置给LeadTools以便验证。当然，这一切LeadTools提供了简单的方法就可以调用，总共只需要3行代码就可以做到这一切。下面，我将通过导入一个既有的Demo来一步步给大家讲解如何运行自己的第一个Android Demo。为了讲解简单，我使用了LeadTools Android SDK 中的示例程序：OCRDemo。 LeadTools Android SDK下载链接：http://leadtools.gcpowertools.com.cn/downloads/ 下载完成后是一个Zip包，请将压缩包解压至任何你喜欢的地方并打开它。解压缩后的目录结构如下图：里面的Examples就都是示例程序啦。 LeadTools 有很多Windows平台下的Demo，非常全面，但是目前开发手机应用的趋势也越来越明显，LeadTools也给大家提供了10个Android的Demo，这篇文章将会教你如何运行第一个Android Demo，我想只有把Demo跑起来了，才能更好地使用LeadTools开发自己的原生Android应用，当然大家也可以把Demo改吧改吧就是自己的原生Android应用。说到Android平台，LeadTools提供了非常全面的支持，但是要将第一个Demo跑起来，License是不可少的。 LeadTools的Android平台，使用的是Runtime License，意思就是需要在程序运行开始时使用代码将得到的License文件读入并设置给LeadTools以便验证。当然，这一切LeadTools提供了简单的方法就可以调用，总共只需要3行代码就可以做到这一切。下面，我将通过导入一个既有的Demo来一步步给大家讲解如何运行自己的第一个Android Demo。为了讲解简单，我使用了LeadTools Android SDK 中的示例程序：OCRDemo。 LeadTools Android SDK下载链接：http://leadtools.gcpowertools.com.cn/downloads/ 下载完成后是一个Zip包，请将压缩包解压至任何你喜欢的地方并打开它。解压缩后的目录结构如下图：里面的Examples就都是示例程序啦。接下来讲解如何将Demo打开并运行它，这就要提到两个Android开发环境了，目前很多人在使用的是Eclipse+ADT+Android SDK环境，但是新出的Android Studio也非常好用，我将就这两个开发工具分别讲解。一、Android Studio 打开Android Studio会看到这个对话框：因为LeadTools所带的示例都是Eclipse格式，所以我们选择Import Non-Android Studio Project。之后弹出对话框，找到我们解压缩的Example文件夹，选择OCRDemo：点击OK，之后选择导入的路径，可以看到Android Studio已经识别出是Eclipse了：选择Next，弹出选项框：这三个选项的意思： 1、 Replace jars with dependencies, when possible-是否在可能的情况下将jar包替换为依赖。 2、 Replace library sources with dependencies, when possible-是否在可能的情况下将library 源替换为依赖。 3、 Create Gradle-style (camelCase) module names-创建Gradle样式的模块名称，首字母小写。我的建议是三个选项都勾上，因为LeadTools的OCR Demo使用到了另一个Android Demo：Leadtools.Demos，所以如果你想把另一个也引用进来的话请你三个都勾上，这样Android Studio会自动将引用到的Project都加载进来。点击Finish之后，Android Studio会自动完成所有导入。完成后，弹出开发窗口，这是会显示一个import-summary.txt，内容就是导入的所有细节。一旦打开开发窗口，就开始了第一次编译，但少年们不要太着急，编译是有错误的：可以看到原因是资源名重名了，问题出现在这两个AndroidManifest.xml中： leadToolsDemos.AndroidManifest.xml ocrDemo.AndroidManifest.xml 为什么会有两个Project，原因就是我刚才说的那样，OCRDemo引用了leadTools.Demos，所以Android Studio会把这两个都加载进来。如何修改呢，简单办法就是将leadToolsDemos.AndroidManifest.xml中的android:icon="@drawable/ic_lead" 这一句删掉。删掉之后，重新编译，成功！二、Eclipse导入 Eclipse相对比较简单，因为原始Demo就是基于Eclipse的。打开File---Import导入。选择Existing Android Code Into Workspace，点击Next。浏览找到LeadTools Android 解压缩路径，选择Examples下的Android即可，这样会把Android目录下的Demo都导入进来。导入之后Eclipse会自动编译，等待完成后就可以运行了。但是到这一步为止，这些Demo仍然无法跑起来，因为缺少License文件，这些Demo都需要Runtime License，我想很多同学都在纠结到底如何使用LeadTools的Runtime License，文档是有的，但是不够清楚，而且没有Android版，那么接下来我就给大家讲解如何在Android程序中部署LeadTools Runtime License。三、部署Runtime License LeadTools目前的Android和IOS都是基于V18的，如果大家需要跑Demo那么需要V18的License，这里提醒大家，V18的License跟V19一样，大家下载V18安装包之后选择GetLicense即可获得License。得到License之后，请大家在Leadtools.Demos的res下新建一个raw资源文件夹，然后将得到的License文件放入其中，如下图：之后修改Support.java文件，将你得到的key文件中的DeveloperKey复制到代码中，如下图：并且将licenseFileId填写到上面即可。完成后，重新编译工程，就可以正常的跑Demo了。本文转自 powertoolsteam 51CTO博客，原文链接：http://blog.51cto.com/powertoolsteam/1625020，如需转载请自行联系原作者

2017-11-20

Hadoop MapReduce编程 API入门系列之倒排索引（二十四）

2016-12-12 21:54:04,509 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Initializing JVM Metrics with processName=JobTracker, sessionId= 2016-12-12 21:54:05,166 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-12 21:54:05,169 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-12 21:54:05,477 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 3 2016-12-12 21:54:05,539 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:3 2016-12-12 21:54:05,810 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1000661716_0001 2016-12-12 21:54:06,184 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-12 21:54:06,185 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1000661716_0001 2016-12-12 21:54:06,193 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-12 21:54:06,220 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-12 21:54:06,297 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-12 21:54:06,314 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1000661716_0001_m_000000_0 2016-12-12 21:54:06,374 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 21:54:06,433 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@6b4d160c 2016-12-12 21:54:06,441 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/data/inverseIndex/b.txt:0+35 2016-12-12 21:54:06,515 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-12 21:54:06,516 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-12 21:54:06,517 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-12 21:54:06,517 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-12 21:54:06,517 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-12 21:54:06,544 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-12 21:54:06,567 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-12 21:54:06,567 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-12 21:54:06,567 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-12 21:54:06,568 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 130; bufvoid = 104857600 2016-12-12 21:54:06,568 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214376(104857504); length = 21/6553600 2016-12-12 21:54:06,590 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-12 21:54:06,599 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1000661716_0001_m_000000_0 is done. And is in the process of committing 2016-12-12 21:54:06,631 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map 2016-12-12 21:54:06,631 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1000661716_0001_m_000000_0' done. 2016-12-12 21:54:06,631 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1000661716_0001_m_000000_0 2016-12-12 21:54:06,631 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1000661716_0001_m_000001_0 2016-12-12 21:54:06,637 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 21:54:06,687 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@418b04a5 2016-12-12 21:54:06,691 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/data/inverseIndex/a.txt:0+33 2016-12-12 21:54:06,742 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-12 21:54:06,742 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-12 21:54:06,742 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-12 21:54:06,742 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-12 21:54:06,743 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-12 21:54:06,744 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-12 21:54:06,747 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-12 21:54:06,748 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-12 21:54:06,748 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-12 21:54:06,748 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 128; bufvoid = 104857600 2016-12-12 21:54:06,748 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214376(104857504); length = 21/6553600 2016-12-12 21:54:06,756 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-12 21:54:06,761 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1000661716_0001_m_000001_0 is done. And is in the process of committing 2016-12-12 21:54:06,766 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map 2016-12-12 21:54:06,766 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1000661716_0001_m_000001_0' done. 2016-12-12 21:54:06,766 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1000661716_0001_m_000001_0 2016-12-12 21:54:06,766 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1000661716_0001_m_000002_0 2016-12-12 21:54:06,769 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 21:54:06,797 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@30616f6c 2016-12-12 21:54:06,800 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/data/inverseIndex/c.txt:0+22 2016-12-12 21:54:06,879 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-12 21:54:06,879 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-12 21:54:06,879 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-12 21:54:06,880 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-12 21:54:06,880 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-12 21:54:06,881 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-12 21:54:06,884 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-12 21:54:06,884 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-12 21:54:06,884 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-12 21:54:06,884 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 86; bufvoid = 104857600 2016-12-12 21:54:06,884 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600 2016-12-12 21:54:06,891 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-12 21:54:06,895 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1000661716_0001_m_000002_0 is done. And is in the process of committing 2016-12-12 21:54:06,898 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map 2016-12-12 21:54:06,898 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1000661716_0001_m_000002_0' done. 2016-12-12 21:54:06,899 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1000661716_0001_m_000002_0 2016-12-12 21:54:06,899 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-12 21:54:06,903 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-12 21:54:06,903 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1000661716_0001_r_000000_0 2016-12-12 21:54:06,917 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 21:54:06,948 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@43234903 2016-12-12 21:54:06,954 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@a609d4 2016-12-12 21:54:06,979 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-12 21:54:06,996 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1000661716_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-12 21:54:07,040 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#1 about to shuffle output of map attempt_local1000661716_0001_m_000000_0 decomp: 144 len: 148 to MEMORY 2016-12-12 21:54:07,052 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 144 bytes from map-output for attempt_local1000661716_0001_m_000000_0 2016-12-12 21:54:07,099 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 144, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->144 2016-12-12 21:54:07,103 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#1 about to shuffle output of map attempt_local1000661716_0001_m_000001_0 decomp: 142 len: 146 to MEMORY 2016-12-12 21:54:07,105 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 142 bytes from map-output for attempt_local1000661716_0001_m_000001_0 2016-12-12 21:54:07,105 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 142, inMemoryMapOutputs.size() -> 2, commitMemory -> 144, usedMemory ->286 2016-12-12 21:54:07,110 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#1 about to shuffle output of map attempt_local1000661716_0001_m_000002_0 decomp: 96 len: 100 to MEMORY 2016-12-12 21:54:07,112 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 96 bytes from map-output for attempt_local1000661716_0001_m_000002_0 2016-12-12 21:54:07,112 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 96, inMemoryMapOutputs.size() -> 3, commitMemory -> 286, usedMemory ->382 2016-12-12 21:54:07,113 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-12 21:54:07,114 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 3 / 3 copied. 2016-12-12 21:54:07,115 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs 2016-12-12 21:54:07,130 INFO [org.apache.hadoop.mapred.Merger] - Merging 3 sorted segments 2016-12-12 21:54:07,131 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 3 segments left of total size: 334 bytes 2016-12-12 21:54:07,133 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 3 segments, 382 bytes to disk to satisfy reduce memory limit 2016-12-12 21:54:07,133 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 382 bytes from disk 2016-12-12 21:54:07,134 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-12 21:54:07,134 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 21:54:07,136 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 362 bytes 2016-12-12 21:54:07,136 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 3 / 3 copied. 2016-12-12 21:54:07,144 INFO [org.apache.hadoop.conf.Configuration.deprecation] - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2016-12-12 21:54:07,163 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1000661716_0001_r_000000_0 is done. And is in the process of committing 2016-12-12 21:54:07,166 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 3 / 3 copied. 2016-12-12 21:54:07,166 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1000661716_0001_r_000000_0 is allowed to commit now 2016-12-12 21:54:07,172 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1000661716_0001_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/InverseIndexStepOne/_temporary/0/task_local1000661716_0001_r_000000 2016-12-12 21:54:07,173 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-12 21:54:07,173 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1000661716_0001_r_000000_0' done. 2016-12-12 21:54:07,174 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1000661716_0001_r_000000_0 2016-12-12 21:54:07,174 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-12 21:54:07,189 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1000661716_0001 running in uber mode : false 2016-12-12 21:54:07,191 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-12 21:54:07,193 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1000661716_0001 completed successfully 2016-12-12 21:54:07,223 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 33 File System Counters FILE: Number of bytes read=5146 FILE: Number of bytes written=777798 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=8 Map output records=16 Map output bytes=344 Map output materialized bytes=394 Input split bytes=396 Combine input records=0 Combine output records=0 Reduce input groups=9 Reduce shuffle bytes=394 Reduce input records=16 Reduce output records=9 Spilled Records=32 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1460142080 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=90 File Output Format Counters Bytes Written=150 2016-12-12 21:55:03,523 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Initializing JVM Metrics with processName=JobTracker, sessionId= 2016-12-12 21:55:05,038 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-12 21:55:05,044 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-12 21:55:05,350 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-12 21:55:05,428 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-12 21:55:05,846 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local549789154_0001 2016-12-12 21:55:06,425 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-12 21:55:06,427 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local549789154_0001 2016-12-12 21:55:06,488 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-12 21:55:06,510 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-12 21:55:06,605 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-12 21:55:06,609 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local549789154_0001_m_000000_0 2016-12-12 21:55:06,691 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 21:55:06,728 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@550aaabb 2016-12-12 21:55:06,738 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/InverseIndexStepOne/part-r-00000:0+138 2016-12-12 21:55:06,821 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-12 21:55:06,821 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-12 21:55:06,821 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-12 21:55:06,821 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-12 21:55:06,821 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-12 21:55:06,828 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-12 21:55:06,851 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-12 21:55:06,852 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-12 21:55:06,852 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-12 21:55:06,852 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 138; bufvoid = 104857600 2016-12-12 21:55:06,852 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600 2016-12-12 21:55:06,882 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-12 21:55:06,895 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local549789154_0001_m_000000_0 is done. And is in the process of committing 2016-12-12 21:55:06,919 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map 2016-12-12 21:55:06,920 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local549789154_0001_m_000000_0' done. 2016-12-12 21:55:06,920 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local549789154_0001_m_000000_0 2016-12-12 21:55:06,921 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-12 21:55:06,927 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-12 21:55:06,928 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local549789154_0001_r_000000_0 2016-12-12 21:55:06,948 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 21:55:06,996 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@1c50c5b8 2016-12-12 21:55:07,002 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@311e2a2d 2016-12-12 21:55:07,024 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-12 21:55:07,029 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local549789154_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-12 21:55:07,073 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#1 about to shuffle output of map attempt_local549789154_0001_m_000000_0 decomp: 158 len: 162 to MEMORY 2016-12-12 21:55:07,079 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 158 bytes from map-output for attempt_local549789154_0001_m_000000_0 2016-12-12 21:55:07,154 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 158, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->158 2016-12-12 21:55:07,156 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-12 21:55:07,157 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 21:55:07,158 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-12 21:55:07,173 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 21:55:07,173 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 150 bytes 2016-12-12 21:55:07,175 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 158 bytes to disk to satisfy reduce memory limit 2016-12-12 21:55:07,176 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 162 bytes from disk 2016-12-12 21:55:07,177 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-12 21:55:07,177 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 21:55:07,179 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 150 bytes 2016-12-12 21:55:07,180 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 21:55:07,188 INFO [org.apache.hadoop.conf.Configuration.deprecation] - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2016-12-12 21:55:07,202 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local549789154_0001_r_000000_0 is done. And is in the process of committing 2016-12-12 21:55:07,206 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 21:55:07,206 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local549789154_0001_r_000000_0 is allowed to commit now 2016-12-12 21:55:07,217 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local549789154_0001_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/InverseIndexStepTwo/_temporary/0/task_local549789154_0001_r_000000 2016-12-12 21:55:07,219 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-12 21:55:07,219 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local549789154_0001_r_000000_0' done. 2016-12-12 21:55:07,219 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local549789154_0001_r_000000_0 2016-12-12 21:55:07,223 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-12 21:55:07,431 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local549789154_0001 running in uber mode : false 2016-12-12 21:55:07,433 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-12 21:55:07,435 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local549789154_0001 completed successfully 2016-12-12 21:55:07,453 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 33 File System Counters FILE: Number of bytes read=1072 FILE: Number of bytes written=386015 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=9 Map output records=9 Map output bytes=138 Map output materialized bytes=162 Input split bytes=145 Combine input records=0 Combine output records=0 Reduce input groups=3 Reduce shuffle bytes=162 Reduce input records=9 Reduce output records=3 Spilled Records=18 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=466616320 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=158 File Output Format Counters Bytes Written=121 代码 package zhouls.bigdata.myMapReduce.InverseIndex; import java.io.IOException; import org.apache.commons.lang.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.InputSplit; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.FileSplit; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; /** * 倒排索引步骤一job * * */ public class InverseIndexStepOne { public static class StepOneMapper extends Mapper<LongWritable, Text, Text, LongWritable>{ @Override protected void map(LongWritable key, Text value,Context context) throws IOException, InterruptedException { //拿到一行数据 String line = value.toString(); //切分出各个单词 String[] fields = StringUtils.split(line, " "); //获取这一行数据所在的文件切片 FileSplit inputSplit = (FileSplit) context.getInputSplit(); //从文件切片中获取文件名 String fileName = inputSplit.getPath().getName(); for(String field:fields){ //封装kv输出， k ： hello-->a.txt v: 1 context.write(new Text(field+"-->"+fileName), new LongWritable(1)); } } } public static class StepOneReducer extends Reducer<Text, LongWritable, Text, LongWritable>{ // <hello-->a.txt,{1,1,1....}> @Override protected void reduce(Text key, Iterable<LongWritable> values,Context context) throws IOException, InterruptedException { long counter = 0; for(LongWritable value:values){ counter += value.get(); } context.write(key, new LongWritable(counter)); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf); job.setJarByClass(InverseIndexStepOne.class); job.setMapperClass(StepOneMapper.class); job.setReducerClass(StepOneReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); // FileInputFormat.setInputPaths(job, new Path("hdfs://HadoopMaster:9000/inverseIndex/")); // // //检查一下参数所指定的输出路径是否存在，如果已存在，先删除 // Path output = new Path("hdfs://HadoopMaster:9000/out/InverseIndexStepOne/"); // FileInputFormat.setInputPaths(job, new Path("./data/inverseIndex/")); //检查一下参数所指定的输出路径是否存在，如果已存在，先删除 Path output = new Path("./out/InverseIndexStepOne"); FileSystem fs = FileSystem.get(conf); if(fs.exists(output)){ fs.delete(output, true); } FileOutputFormat.setOutputPath(job, output); System.exit(job.waitForCompletion(true)?0:1); } } package zhouls.bigdata.myMapReduce.InverseIndex; import java.io.IOException; import org.apache.commons.lang.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Mapper.Context; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.Reducer; import zhouls.bigdata.myMapReduce.InverseIndex.InverseIndexStepOne.StepOneMapper; import zhouls.bigdata.myMapReduce.InverseIndex.InverseIndexStepOne.StepOneReducer; public class InverseIndexStepTwo { public static class StepTwoMapper extends Mapper<LongWritable, Text, Text, Text>{ //k: 行起始偏移量 v: {hello-->a.txt 3} @Override protected void map(LongWritable key, Text value,Context context) throws IOException, InterruptedException { String line = value.toString(); String[] fields = StringUtils.split(line, "\t"); String[] wordAndfileName = StringUtils.split(fields[0], "-->"); String word = wordAndfileName[0]; String fileName = wordAndfileName[1]; long count = Long.parseLong(fields[1]); context.write(new Text(word), new Text(fileName+"-->"+count)); //map输出的结果是这个形式： <hello,a.txt-->3> } } public static class StepTwoReducer extends Reducer<Text, Text,Text, Text>{ @Override protected void reduce(Text key, Iterable<Text> values,Context context) throws IOException, InterruptedException { //拿到的数据 <hello,{a.txt-->3,b.txt-->2,c.txt-->1}> String result = ""; for(Text value:values){ result += value + " "; } context.write(key, new Text(result)); //输出的结果就是 k: hello v: a.txt-->3 b.txt-->2 c.txt-->1 } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); //先构造job_one // Job job_one = Job.getInstance(conf); // // job_one.setJarByClass(InverseIndexStepTwo.class); // job_one.setMapperClass(StepOneMapper.class); // job_one.setReducerClass(StepOneReducer.class); //...... //构造job_two Job job_tow = Job.getInstance(conf); job_tow.setJarByClass(InverseIndexStepTwo.class); job_tow.setMapperClass(StepTwoMapper.class); job_tow.setReducerClass(StepTwoReducer.class); job_tow.setOutputKeyClass(Text.class); job_tow.setOutputValueClass(Text.class); // FileInputFormat.setInputPaths(job_tow, new Path("hdfs://HadoopMaster:9000/out/InverseIndexStepOne/")); // // //检查一下参数所指定的输出路径是否存在，如果已存在，先删除 // Path output = new Path("hdfs://HadoopMaster:9000/out/InverseIndexStepTwo/"); FileInputFormat.setInputPaths(job_tow, new Path("./out/InverseIndexStepOne")); //检查一下参数所指定的输出路径是否存在，如果已存在，先删除 Path output = new Path("./out/InverseIndexStepTwo"); FileSystem fs = FileSystem.get(conf); if(fs.exists(output)){ fs.delete(output, true); } FileOutputFormat.setOutputPath(job_tow, output); //先提交job_one执行 // boolean one_result = job_one.waitForCompletion(true); // if(one_result){ System.exit(job_tow.waitForCompletion(true)?0:1); // } } } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6166185.html，如需转载请自行联系原作者

2017-11-20

Hadoop MapReduce编程 API入门系列之MapReduce多种输入格式（十七）

代码 1 package zhouls.bigdata.myMapReduce.ScoreCount; 2 3 import java.io.DataInput; 4 import java.io.DataOutput; 5 import java.io.IOException; 6 import org.apache.hadoop.io.WritableComparable; 7 /** 8 * 学习成绩读写类 9 * 数据格式参考：19020090017 小讲 90 99 100 89 95 10 * @author Bertron 11 * 需要自定义一个 ScoreWritable 类实现 WritableComparable 接口，将学生各门成绩封装起来。 12 */ 13 public class ScoreWritable implements WritableComparable< Object > {//其实这里，跟TVPlayData一样的 14 // 注意： Hadoop通过Writable接口实现的序列化机制，不过没有提供比较功能，所以和java中的Comparable接口合并，提供一个接口WritableComparable。（自定义比较） 15 // Writable接口提供两个方法(write和readFields)。 16 17 18 private float Chinese; 19 private float Math; 20 private float English; 21 private float Physics; 22 private float Chemistry; 23 24 25 // 问：这里我们自己编程时，是一定要创建一个带有参的构造方法，为什么还要显式的写出来一个带无参的构造方法呢？ 26 // 答：构造器其实就是构造对象实例的方法，无参数的构造方法是默认的，但是如果你创造了一个带有参数的构造方法，那么无参的构造方法必须显式的写出来，否则会编译失败。 27 28 public ScoreWritable(){}//java里的无参构造函数，是用来在创建对象时初始化对象 29 //在hadoop的每个自定义类型代码里，好比，现在的ScoreWritable，都必须要写无参构造函数。 30 31 32 //问：为什么我们在编程的时候，需要创建一个带有参的构造方法？ 33 //答:就是能让赋值更灵活。构造一般就是初始化数值，你不想别人用你这个类的时候每次实例化都能用另一个构造动态初始化一些信息么(当然没有需要额外赋值就用默认的)。 34 35 public ScoreWritable(float Chinese,float Math,float English,float Physics,float Chemistry){//java里的有参构造函数,是用来在创建对象时初始化对象 36 this.Chinese = Chinese; 37 this.Math = Math; 38 this.English = English; 39 this.Physics = Physics; 40 this.Chemistry = Chemistry; 41 } 42 43 //问：其实set和get方法，这两个方法只是类中的setxxx和getxxx方法的总称， 44 // 那么，为什么在编程时，有set和set***两个，只有get***一个呢？ 45 46 public void set(float Chinese,float Math,float English,float Physics,float Chemistry){ 47 this.Chinese = Chinese;//即float Chinese赋值给private float Chinese; 48 this.Math = Math; 49 this.English = English; 50 this.Physics = Physics; 51 this.Chemistry = Chemistry; 52 } 53 // public float get(float Chinese,float Math,float English,float Physics,float Chemistry){因为这是错误的，所以对于set可以分开，get只能是get*** 54 // return Chinese; 55 // return Math; 56 // return English; 57 // return Physics; 58 // return Chemistry; 59 // } 60 61 62 public float getChinese() {//拿值，得返回，所以需有返回类型float 63 return Chinese; 64 } 65 public void setChinese(float Chinese){//设值，不需，所以空返回类型 66 this.Chinese = Chinese; 67 } 68 public float getMath() {//拿值 69 return Math; 70 } 71 public void setMath(float Math){//设值 72 this.Math = Math; 73 } 74 public float getEnglish() {//拿值 75 return English; 76 } 77 public void setEnglish(float English){//设值 78 this.English = English; 79 } 80 public float getPhysics() {//拿值 81 return Physics; 82 } 83 public void setPhysics(float Physics){//设值 84 this.Physics = Physics; 85 } 86 public float getChemistry() {//拿值 87 return Chemistry; 88 } 89 public void setChemistry(float Chemistry) {//拿值 90 this.Chemistry = Chemistry; 91 } 92 93 // 实现WritableComparable的readFields()方法 94 // 对象不能传输的，需要转化成字节流！ 95 // 将对象转换为字节流并写入到输出流out中是序列化，write 的过程（最好记!!!） 96 // 从输入流in中读取字节流反序列化为对象是反序列化，readFields的过程（最好记!!!） 97 public void readFields(DataInput in) throws IOException {//拿代码来说的话，对象就是比如Chinese、Math。。。。 98 Chinese = in.readFloat();//因为，我们这里的对象是float类型，所以是readFloat() 99 Math = in.readFloat(); 100 English = in.readFloat();//注意:反序列化里，需要生成对象对吧，所以，是用到的是get那边对象 101 Physics = in.readFloat(); 102 Chemistry = in.readFloat(); 103 // in.readByte() 104 // in.readChar() 105 // in.readDouble() 106 // in.readLine() 107 // in.readFloat() 108 // in.readLong() 109 // in.readShort() 110 } 111 112 // 实现WritableComparable的write()方法，以便该数据能被序列化后完成网络传输或文件输出 113 // 将对象转换为字节流并写入到输出流out中是序列化，write 的过程（最好记!!!） 114 // 从输入流in中读取字节流反序列化为对象是反序列化，readFields的过程（最好记!!!） 115 public void write(DataOutput out) throws IOException {//拿代码来说的话，对象就是比如Chinese、Math。。。。 116 out.writeFloat(Chinese);//因为，我们这里的对象是float类型，所以是writeFloat() 117 out.writeFloat(Math); 118 out.writeFloat(English);//注意:序列化里，需要对象对吧，所以，用到的是set那边的对象 119 out.writeFloat(Physics); 120 out.writeFloat(Chemistry); 121 // out.writeByte() 122 // out.writeChar() 123 // out.writeDouble() 124 // out.writeFloat() 125 // out.writeLong() 126 // out.writeShort() 127 // out.writeUTF() 128 } 129 130 public int compareTo(Object o) {//java里的比较，Java String.compareTo() 131 return 0; 132 } 133 134 135 // Hadoop中定义了两个序列化相关的接口：Writable 接口和 Comparable 接口，这两个接口可以合并成一个接口 WritableComparable。 136 // Writable 接口中定义了两个方法，分别为write(DataOutput out)和readFields(DataInput in) 137 // 所有实现了Comparable接口的对象都可以和自身相同类型的对象比较大小 138 139 140 // Hadoop中定义了两个序列化相关的接口：Writable 接口和 Comparable 接口，这两个接口可以合并成一个接口 WritableComparable。 141 // Writable 接口中定义了两个方法，分别为write(DataOutput out)和readFields(DataInput in) 142 // 所有实现了Comparable接口的对象都可以和自身相同类型的对象比较大小 143 144 145 // 源码是 146 // package java.lang; 147 // import java.util.*; 148 // public interface Comparable { 149 // /** 150 // * 将this对象和对象o进行比较，约定：返回负数为小于，零为大于，整数为大于 151 // */ 152 // public int compareTo(T o); 153 // } 154 155 } 1 package zhouls.bigdata.myMapReduce.ScoreCount; 2 3 import java.io.IOException; 4 import org.apache.hadoop.conf.Configuration; 5 import org.apache.hadoop.fs.FSDataInputStream; 6 import org.apache.hadoop.fs.FileSystem; 7 import org.apache.hadoop.fs.Path; 8 import org.apache.hadoop.io.Text; 9 import org.apache.hadoop.mapreduce.InputSplit; 10 import org.apache.hadoop.mapreduce.JobContext; 11 import org.apache.hadoop.mapreduce.RecordReader; 12 import org.apache.hadoop.mapreduce.TaskAttemptContext; 13 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 14 import org.apache.hadoop.mapreduce.lib.input.FileSplit; 15 import org.apache.hadoop.util.LineReader; 16 /** 17 * 自定义学生成绩读写InputFormat 18 * 数据格式参考：19020090017 小讲 90 99 100 89 95 19 * @author Bertron 20 */ 21 22 //其实这个程序，就是在实现InputFormat接口，TVPlayInputFormat是InputFormat接口的实现类 23 //比如 ScoreInputFormat extends FileInputFormat implements InputFormat。 24 25 //问：自定义输入格式 ScoreInputFormat 类，首先继承 FileInputFormat，然后分别重写 isSplitable() 方法和 createRecordReader() 方法。 26 27 public class ScoreInputFormat extends FileInputFormat<Text,ScoreWritable > {//自定义数据输入格式，其实这都是模仿源码的！可以去看 28 29 // 线路是： boolean isSplitable() -> RecordReader<Text,ScoreWritable> createRecordReader() -> ScoreRecordReader extends RecordReader<Text, ScoreWritable > 30 31 @Override 32 protected boolean isSplitable(JobContext context, Path filename) {//这是InputFormat的isSplitable方法 33 //isSplitable方法就是是否要切分文件，这个方法显示如果是压缩文件就不切分，非压缩文件就切分。 34 // 如果不允许分割，则isSplitable==false，则将第一个block、文件目录、开始位置为0，长度为整个文件的长度封装到一个InputSplit，加入splits中 35 // 如果文件长度不为0且支持分割，则isSplitable==true,获取block大小，默认是64MB 36 return false; //整个文件封装到一个InputSplit 37 //要么就是return true; //切分64MB大小的一块一块，再封装到InputSplit 38 } 39 40 @Override 41 public RecordReader<Text,ScoreWritable> createRecordReader(InputSplit inputsplit,TaskAttemptContext context) throws IOException, InterruptedException { 42 // RecordReader<k1, v1>是返回类型,返回的RecordReader对象的封装 43 // createRecordReader是方法,在这里是，ScoreInputFormat.createRecordReader。ScoreInputFormat是InputFormat类的实例 44 // InputSplit input和TaskAttemptContext context是传入参数 45 46 // isSplitable(),如果是压缩文件就不切分，整个文件封装到一个InputSplit 47 // isSplitable(),如果是非压缩文件就切，切分64MB大小的一块一块，再封装到InputSplit 48 49 //这里默认是系统实现的的RecordReader，按行读取，下面我们自定义这个类ScoreRecordReader。 50 //类似与Excel、WeiBo、TVPlayData代码写法 51 return new ScoreRecordReader();//新建一个ScoreRecordReader实例，所有才有了上面RecordReader<Text,ScoreWritable>，所以才如下ScoreRecordReader，写我们自己的 52 } 53 54 55 //RecordReader中的两个参数分别填写我们期望返回的key/value类型，我们期望key为Text类型，value为ScoreWritable类型封装学生所有成绩 56 public static class ScoreRecordReader extends RecordReader<Text, ScoreWritable > {//RecordReader<k1, v1>是一个整体 57 public LineReader in;//行读取器 58 public Text line;//每行数据类型 59 public Text lineKey;//自定义key类型，即k1 60 public ScoreWritable lineValue;//自定义value类型，即v1 61 62 @Override 63 public void close() throws IOException {//关闭输入流 64 if(in !=null){ 65 in.close(); 66 } 67 } 68 @Override 69 public Text getCurrentKey() throws IOException, InterruptedException {//获取当前的key,即CurrentKey 70 return lineKey;//返回类型是Text,即Text lineKey 71 } 72 @Override 73 public ScoreWritable getCurrentValue() throws IOException,InterruptedException {//获取当前的Value，即CurrentValue 74 return lineValue;//返回类型是ScoreWritable,即ScoreWritable lineValue 75 } 76 @Override 77 public float getProgress() throws IOException, InterruptedException {//获取进程，即Progress 78 return 0;//返回类型是float,即float 0 79 } 80 @Override 81 public void initialize(InputSplit input, TaskAttemptContext context) throws IOException, InterruptedException {//初始化，都是模板 82 FileSplit split=(FileSplit)input; 83 Configuration job=context.getConfiguration(); 84 Path file=split.getPath(); 85 FileSystem fs=file.getFileSystem(job); 86 87 FSDataInputStream filein=fs.open(file); 88 in=new LineReader(filein,job);//输入流in 89 line=new Text();//每行数据类型 90 lineKey=new Text();//自定义key类型，即k1。//新建一个Text实例作为自定义格式输入的key 91 lineValue = new ScoreWritable();//自定义value类型，即v1。//新建一个TVPlayData实例作为自定义格式输入的value 92 } 93 94 //此方法读取每行数据，完成自定义的key和value 95 @Override 96 public boolean nextKeyValue() throws IOException, InterruptedException {//这里面，才是篡改的重点 97 int linesize=in.readLine(line);//line是每行数据，我们这里用到的是in.readLine(str)这个构造函数，默认读完读到文件末尾。其实这里有三种。 98 99 // 是SplitLineReader.readLine -> SplitLineReader extends LineReader -> org.apache.hadoop.util.LineReader 100 101 // in.readLine(str)//这个构造方法执行时，会首先将value原来的值清空。默认读完读到文件末尾 102 // in.readLine(str, maxLineLength)//只读到maxLineLength行 103 // in.readLine(str, maxLineLength, maxBytesToConsume)//这个构造方法来实现不清空，前面读取的行的值 104 105 if(linesize==0) return false; 106 107 108 String[] pieces = line.toString().split("\\s+");//解析每行数据 109 //因为，我们这里是。默认读完读到文件末尾。line是Text类型。pieces是String[]，即String数组。 110 111 if(pieces.length != 7){ 112 throw new IOException("Invalid record received"); 113 } 114 //将学生的每门成绩转换为 float 类型 115 float a,b,c,d,e; 116 try{ 117 a = Float.parseFloat(pieces[2].trim());//将String类型，如pieces[2]转换成，float类型，给a 118 b = Float.parseFloat(pieces[3].trim()); 119 c = Float.parseFloat(pieces[4].trim()); 120 d = Float.parseFloat(pieces[5].trim()); 121 e = Float.parseFloat(pieces[6].trim()); 122 }catch(NumberFormatException nfe){ 123 throw new IOException("Error parsing floating poing value in record"); 124 } 125 lineKey.set(pieces[0]+"\t"+pieces[1]);//完成自定义key数据 126 lineValue.set(a, b, c, d, e);//封装自定义value数据 127 // 或者写 128 // lineValue.set(Float.parseFloat(pieces[2].trim()),Float.parseFloat(pieces[3].trim()),Float.parseFloat(pieces[4].trim()), 129 // Float.parseFloat(pieces[5].trim()),Float.parseFloat(pieces[6].trim())); 130 131 // pieces[0] pieces[1] pieces[2] ... pieces[6] 132 // 19020090040 秦心芯 123 131 100 95 100 133 // 19020090006 李磊 99 92 100 90 100 134 // 19020090017 唐一建 90 99 100 89 95 135 // 19020090031 曾丽丽 100 99 97 79 96 136 // 19020090013 罗开俊 105 115 94 45 100 137 // 19020090039 周世海 114 116 93 31 97 138 // 19020090020 王正伟 109 98 88 47 99 139 // 19020090025 谢瑞彬 94 120 100 50 73 140 // 19020090007 于微 89 78 100 66 99 141 // 19020090012 刘小利 87 82 89 71 99 142 143 144 145 return true; 146 } 147 } 148 } 1 package zhouls.bigdata.myMapReduce.ScoreCount; 2 3 4 import java.io.IOException; 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.conf.Configured; 7 import org.apache.hadoop.fs.FileSystem; 8 import org.apache.hadoop.fs.Path; 9 import org.apache.hadoop.io.Text; 10 import org.apache.hadoop.mapreduce.Job; 11 import org.apache.hadoop.mapreduce.Mapper; 12 import org.apache.hadoop.mapreduce.Reducer; 13 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 14 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 15 import org.apache.hadoop.util.Tool; 16 import org.apache.hadoop.util.ToolRunner; 17 /** 18 * 学生成绩统计Hadoop程序 19 * 数据格式参考：19020090017 小讲 90 99 100 89 95 20 * @author HuangBQ 21 */ 22 public class ScoreCount extends Configured implements Tool{ 23 public static class ScoreMapper extends Mapper<Text,ScoreWritable,Text,ScoreWritable>{ 24 @Override 25 protected void map(Text key, ScoreWritable value, Context context)throws IOException, InterruptedException{ 26 context.write(key, value);//写入key是k2,value是v2 27 // context.write(new Text(key), new ScoreWritable(value));等价 28 } 29 } 30 31 public static class ScoreReducer extends Reducer<Text,ScoreWritable,Text,Text>{ 32 private Text text = new Text(); 33 protected void reduce(Text Key, Iterable< ScoreWritable > Values, Context context)throws IOException, InterruptedException{ 34 float totalScore=0.0f; 35 float averageScore = 0.0f; 36 for(ScoreWritable ss:Values){ 37 totalScore +=ss.getChinese()+ss.getMath()+ss.getEnglish()+ss.getPhysics()+ss.getChemistry(); 38 averageScore +=totalScore/5; 39 } 40 text.set(totalScore+"\t"+averageScore); 41 context.write(Key, text);//写入Key是k3,text是v3 42 // context.write(new Text(Key),new Text(text));等价 43 } 44 } 45 46 47 public int run(String[] args) throws Exception{ 48 Configuration conf = new Configuration();//读取配置文件 49 50 Path mypath = new Path(args[1]); 51 FileSystem hdfs = mypath.getFileSystem(conf);//创建输出路径 52 if (hdfs.isDirectory(mypath)) 53 { 54 hdfs.delete(mypath, true); 55 } 56 57 Job job = new Job(conf, "ScoreCount");//新建任务 58 job.setJarByClass(ScoreCount.class);//设置主类 59 60 FileInputFormat.addInputPath(job, new Path(args[0]));// 输入路径 61 FileOutputFormat.setOutputPath(job, new Path(args[1]));// 输出路径 62 63 job.setMapperClass(ScoreMapper.class);// Mapper 64 job.setReducerClass(ScoreReducer.class);// Reducer 65 66 job.setMapOutputKeyClass(Text.class);// Mapper key输出类型 67 job.setMapOutputValueClass(ScoreWritable.class);// Mapper value输出类型 68 69 job.setInputFormatClass(ScoreInputFormat.class);//设置自定义输入格式 70 71 job.waitForCompletion(true); 72 return 0; 73 } 74 75 76 77 public static void main(String[] args) throws Exception{ 78 // String[] args0 = 79 // { 80 // "hdfs://HadoopMaster:9000/score/score.txt", 81 // "hdfs://HadoopMaster:9000/out/score/" 82 // }; 83 84 String[] args0 = 85 { 86 "./data/score/score.txt", 87 "./out/score/" 88 }; 89 90 int ec = ToolRunner.run(new Configuration(), new ScoreCount(), args0); 91 System.exit(ec); 92 } 93 } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6165667.html，如需转载请自行联系原作者

2017-11-20

Hadoop MapReduce编程 API入门系列之二次排序（十六）

2016-12-12 17:04:32,012 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Initializing JVM Metrics with processName=JobTracker, sessionId= 2016-12-12 17:04:33,056 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-12 17:04:33,059 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-12 17:04:33,083 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-12 17:04:33,161 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-12 17:04:33,562 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1173601391_0001 2016-12-12 17:04:34,242 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-12 17:04:34,244 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1173601391_0001 2016-12-12 17:04:34,247 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-12 17:04:34,264 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-12 17:04:34,371 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-12 17:04:34,373 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1173601391_0001_m_000000_0 2016-12-12 17:04:34,439 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 17:04:34,667 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@65bb90dc 2016-12-12 17:04:34,676 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/data/secondarySort/secondarySort.txt:0+120 2016-12-12 17:04:34,762 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-12 17:04:34,763 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-12 17:04:34,763 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-12 17:04:34,763 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-12 17:04:34,763 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-12 17:04:34,771 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-12 17:04:34,789 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-12 17:04:34,789 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-12 17:04:34,789 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-12 17:04:34,789 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 216; bufvoid = 104857600 2016-12-12 17:04:34,790 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214328(104857312); length = 69/6553600 2016-12-12 17:04:34,809 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-12 17:04:34,818 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1173601391_0001_m_000000_0 is done. And is in the process of committing 2016-12-12 17:04:34,838 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map 2016-12-12 17:04:34,838 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1173601391_0001_m_000000_0' done. 2016-12-12 17:04:34,838 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1173601391_0001_m_000000_0 2016-12-12 17:04:34,839 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-12 17:04:34,846 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-12 17:04:34,846 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1173601391_0001_r_000000_0 2016-12-12 17:04:34,864 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-12 17:04:34,950 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@59b59452 2016-12-12 17:04:34,954 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@73d5cf65 2016-12-12 17:04:34,974 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-12 17:04:35,011 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1173601391_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-12 17:04:35,048 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#1 about to shuffle output of map attempt_local1173601391_0001_m_000000_0 decomp: 254 len: 258 to MEMORY 2016-12-12 17:04:35,060 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 254 bytes from map-output for attempt_local1173601391_0001_m_000000_0 2016-12-12 17:04:35,123 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 254, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->254 2016-12-12 17:04:35,125 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-12 17:04:35,126 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 17:04:35,126 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-12 17:04:35,136 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 17:04:35,137 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 244 bytes 2016-12-12 17:04:35,139 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 254 bytes to disk to satisfy reduce memory limit 2016-12-12 17:04:35,139 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 258 bytes from disk 2016-12-12 17:04:35,140 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-12 17:04:35,141 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-12 17:04:35,142 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 244 bytes 2016-12-12 17:04:35,143 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 17:04:35,150 INFO [org.apache.hadoop.conf.Configuration.deprecation] - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 2016-12-12 17:04:35,158 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1173601391_0001_r_000000_0 is done. And is in the process of committing 2016-12-12 17:04:35,160 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-12 17:04:35,160 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1173601391_0001_r_000000_0 is allowed to commit now 2016-12-12 17:04:35,166 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1173601391_0001_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/secondarySort/_temporary/0/task_local1173601391_0001_r_000000 2016-12-12 17:04:35,167 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-12 17:04:35,167 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1173601391_0001_r_000000_0' done. 2016-12-12 17:04:35,167 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1173601391_0001_r_000000_0 2016-12-12 17:04:35,168 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-12 17:04:35,248 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1173601391_0001 running in uber mode : false 2016-12-12 17:04:35,249 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-12 17:04:35,251 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1173601391_0001 completed successfully 2016-12-12 17:04:35,271 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 33 File System Counters FILE: Number of bytes read=1186 FILE: Number of bytes written=394623 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=18 Map output records=18 Map output bytes=216 Map output materialized bytes=258 Input split bytes=145 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=258 Reduce input records=18 Reduce output records=18 Spilled Records=36 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=534773760 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=120 File Output Format Counters Bytes Written=115 代码 IntPair.java package zhouls.bigdata.myMapReduce.SecondarySort; import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import org.apache.hadoop.io.WritableComparable; //第一步：自定义IntPair类，将示例数据中的key/value封装成一个整体作为Key，同时实现 WritableComparable 接口并重写其方法。 /** * 自己定义的key类应该实现WritableComparable接口 */ public class IntPair implements WritableComparable<IntPair>{//类似对应于如TextPair int first;//第一个成员变量 int second;//第二个成员变量 public void set(int left, int right){//赋值 first = left; second = right; } public int getFirst(){//读值 return first; } public int getSecond(){//读值 return second; } //反序列化，从流中的二进制转换成IntPair public void readFields(DataInput in) throws IOException{ first = in.readInt(); second = in.readInt(); } //序列化，将IntPair转化成使用流传送的二进制 public void write(DataOutput out) throws IOException{ out.writeInt(first); out.writeInt(second); } //key的比较 public int compareTo(IntPair o){ // TODO Auto-generated method stub if (first != o.first){ return first < o.first ? -1 : 1; }else if (second != o.second) { return second < o.second ? -1 : 1; }else { return 0; } } @Override public int hashCode(){ return first * 157 + second; } @Override public boolean equals(Object right){ if (right == null) return false; if (this == right) return true; if (right instanceof IntPair){ IntPair r = (IntPair) right; return r.first == first && r.second == second; }else{ return false; } } } SecondarySort.java package zhouls.bigdata.myMapReduce.SecondarySort; import zhouls.bigdata.myMapReduce.Join.JoinRecordAndStationName; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.io.WritableComparable; import org.apache.hadoop.io.WritableComparator; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Partitioner; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; /* SecondarySort内容是 40 20 40 10 40 30 40 5 40 1 30 30 30 20 30 10 30 1 20 20 20 10 20 1 50 50 50 40 50 30 50 20 50 10 50 1 */ public class SecondarySort extends Configured implements Tool{ // 自定义map public static class Map extends Mapper<LongWritable, Text, IntPair, IntWritable>{ private final IntPair intkey = new IntPair(); private final IntWritable intvalue = new IntWritable(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{ String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); int left = 0; int right = 0; if (tokenizer.hasMoreTokens()){ left = Integer.parseInt(tokenizer.nextToken()); if (tokenizer.hasMoreTokens()) right = Integer.parseInt(tokenizer.nextToken()); intkey.set(left, right);//设为k2 intvalue.set(right);//设为v2 context.write(intkey,intvalue);//写入intkeyk2,intvalue是v2 // context.write(new IntPair(intkey),new IntWritable(intvalue));等价 } } } //第二步：自定义分区函数类FirstPartitioner，根据 IntPair 中的first实现分区。 /** * 分区函数类。根据first确定Partition。 */ public static class FirstPartitioner extends Partitioner< IntPair, IntWritable>{ @Override public int getPartition(IntPair key, IntWritable value,int numPartitions){ return Math.abs(key.getFirst() * 127) % numPartitions; } } //第三步：自定义 SortComparator 实现 IntPair 类中的first和second排序。本课程中没有使用这种方法，而是使用 IntPair 中的compareTo()方法实现的。 //第四步：自定义 GroupingComparator 类，实现分区内的数据分组。 /** *继承WritableComparator */ public static class GroupingComparator extends WritableComparator{ protected GroupingComparator(){ super(IntPair.class, true); } @Override //Compare two WritableComparables. public int compare(WritableComparable w1, WritableComparable w2){ IntPair ip1 = (IntPair) w1; IntPair ip2 = (IntPair) w2; int l = ip1.getFirst(); int r = ip2.getFirst(); return l == r ? 0 : (l < r ? -1 : 1); } } // 自定义reduce public static class Reduce extends Reducer<IntPair, IntWritable, Text, IntWritable>{ private final Text left = new Text(); public void reduce(IntPair key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException{ left.set(Integer.toString(key.getFirst()));//设为k3 for (IntWritable val : values){ context.write(left, val);//写入left是k3,val是v3 // context.write(new Text(left),new IntWritable(val));等价 } } } public int run(String[] args)throws Exception{ // TODO Auto-generated method stub Configuration conf = new Configuration(); Path mypath=new Path(args[1]); FileSystem hdfs = mypath.getFileSystem(conf); if (hdfs.isDirectory(mypath)){ hdfs.delete(mypath, true); } Job job = new Job(conf, "secondarysort"); job.setJarByClass(SecondarySort.class); FileInputFormat.setInputPaths(job, new Path(args[0]));//输入路径 FileOutputFormat.setOutputPath(job, new Path(args[1]));//输出路径 job.setMapperClass(Map.class);// Mapper job.setReducerClass(Reduce.class);// Reducer //job.setNumReducerTask(3); job.setPartitionerClass(FirstPartitioner.class);// 分区函数 //job.setSortComparatorClass(KeyComparator.Class);//本课程并没有自定义SortComparator，而是使用IntPair自带的排序 job.setGroupingComparatorClass(GroupingComparator.class);// 分组函数 job.setMapOutputKeyClass(IntPair.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); return job.waitForCompletion(true) ? 0 : 1; } /** * @param args * @throws Exception */ public static void main(String[] args) throws Exception{ // TODO Auto-generated method stub // String[] args0={"hdfs://HadoopMaster:9000/secondarySort/secondarySort.txt", // "hdfs://HadoopMaster:9000/out/secondarySort"}; String[] args0={"./data/secondarySort/secondarySort.txt", "./out/secondarySort"}; int ec =ToolRunner.run(new Configuration(),new SecondarySort(),args0); System.exit(ec); } } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6165256.html，如需转载请自行联系原作者

2017-11-19

Hadoop入门进阶课程12--Flume介绍、安装与应用案例

本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接，博主为石山园，博客地址为http://www.cnblogs.com/shishanyuan。该系列课程是应邀实验楼整理编写的，这里需要赞一下实验楼提供了学习的新方式，可以边看博客边上机实验，课程地址为https://www.shiyanlou.com/courses/237 【注】该系列所使用到安装包、测试数据和代码均可在百度网盘下载，具体地址为http://pan.baidu.com/s/10PnDs，下载该PDF文件 1、搭建环境部署节点操作系统为CentOS，防火墙和SElinux禁用，创建了一个shiyanlou用户并在系统根目录下创建/app目录，用于存放Hadoop等组件运行包。因为该目录用于安装hadoop等组件程序，用户对shiyanlou必须赋予rwx权限（一般做法是root用户在根目录下创建/app目录，并修改该目录拥有者为shiyanlou(chown–R shiyanlou:shiyanlou /app）。 Hadoop搭建环境： l虚拟机操作系统：CentOS6.664位，单核，1G内存 lJDK：1.7.0_55 64位 lHadoop：1.1.2 2、Flume介绍 Flume是Cloudera提供的日志收集系统，Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。Flume是一个分布式、可靠和高可用的海量日志采集、聚合和传输的系统。 Flume具有Reliability、Scalability、Manageability和Extensibility特点： 1.Reliability：Flume提供3中数据可靠性选项，包括End-to-end、Store on failure和Best effort。其中End-to-end使用了磁盘日志和接受端Ack的方式，保证Flume接受到的数据会最终到达目的。Store on failure在目的不可用的时候，数据会保持在本地硬盘。和End-to-end不同的是，如果是进程出现问题，Store on failure可能会丢失部分数据。Best effort不做任何QoS保证。 2.Scalability：Flume的3大组件：collector、master和storage tier都是可伸缩的。需要注意的是，Flume中对事件的处理不需要带状态，它的Scalability可以很容易实现。 3.Manageability：Flume利用ZooKeeper和gossip，保证配置数据的一致性、高可用。同时，多Master，保证Master可以管理大量的节点。 4.Extensibility：基于Java，用户可以为Flume添加各种新的功能，如通过继承Source，用户可以实现自己的数据接入方式，实现Sink的子类，用户可以将数据写往特定目标，同时，通过SinkDecorator，用户可以对数据进行一定的预处理。 2.1Flume架构上图的Flume的架构中最重要的抽象是data flow（数据流），data flow描述了数据从产生，传输、处理并最终写入目标的一条路径（在上图中，实线描述了data flow）。Agent用于采集数据，agent是flume中产生数据流的地方，同时，agent会将产生的数据流传输到collector。对应的，collector用于对数据进行聚合，往往会产生一个更大的流。 Flume提供了从console（控制台）、RPC（Thrift-RPC）、text（文件）、tail（UNIX tail）、syslog（syslog日志系统，支持TCP和UDP等2种模式），exec（命令执行）等数据源上收集数据的能力。同时，Flume的数据接受方，可以是console（控制台）、text（文件）、dfs（HDFS文件）、RPC（Thrift-RPC）和syslogTCP（TCP syslog日志系统）等。其中，收集数据有2种主要工作模式，如下： 1.Push Sources：外部系统会主动地将数据推送到Flume中，如RPC、syslog。 2.Polling Sources：Flume到外部系统中获取数据，一般使用轮询的方式，如text和exec。注意，在Flume中，agent和collector对应，而source和sink对应。Source和sink强调发送、接受方的特性（如数据格式、编码等），而agent和collector关注功能。 2.2Flume管理方式 Flume Master用于管理数据流的配置，如下图。为了保证可扩展性，Flume采用了多Master的方式。为了保证配置数据的一致性，Flume引入了ZooKeeper，用于保存配置数据，ZooKeeper本身可保证配置数据的一致性和高可用，另外，在配置数据发生变化时，ZooKeeper可以通知Flume Master节点。 Flume Master间使用gossip协议同步数据。 3、安装部署Flume 3.1Flume部署过程 3.1.1下载Flume 可以到apache基金flume官网http://flume.apache.org/download.html，选择镜像下载地址http://mirrors.hust.edu.cn/apache/flume/下载一个稳定版本，如下图所示下载flume-1.5.2-bin.tar.gz：也可以在/home/shiyanlou/install-pack目录中找到该安装包，解压该安装包并把该安装包复制到/app目录中 cd /home/shiyanlou/install-pack tar -xzf flume-1.5.2-bin.tar.gz mv apache-flume-1.5.2-bin /app/flume-1.5.2 3.1.2设置/etc/profile参数编辑/etc/profile文件，声明flume的home路径和在path加入bin的路径： export FLUME_HOME=/app/flume-1.5.2 export FLUME_CONF_DIR=$FLUME_HOME/conf export PATH=$PATH:$FLUME_HOME/bin 编译配置文件/etc/profile，并确认生效 source /etc/profile echo $PATH 3.1.3设置flume-env.sh配置文件在$FLUME_HOME/conf下复制改名flume-env.sh.template为flume-env.sh，修改conf/flume-env.sh配置文件 cd /app/flume-1.5.2/conf cp flume-env.sh.template flume-env.sh sudo vi flume-env.sh 修改配置文件内容： JAVA_HOME=/app/lib/jdk1.7.0_55 JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote" 3.2部署验证 3.2.1验证安装 1.修改flume-conf配置文件在$FLUME_HOME/conf目录下修改flume-conf.properties.template文件，复制并改名为flume-conf， cd /app/flume-1.5.2/conf cp flume-conf.properties.template flume-conf.properties sudo vi flume-conf.properties 修改flume-conf配置文件内容 # The configuration file needs to define the sources, the channels and the sinks. # Sources, channels and sinks are defined per agent, in this case called 'a1' a1.sources = r1 a1.sinks = k1 a1.channels = c1 # For each one of the sources, the type is defined a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 #The channel can be defined as follows. a1.sources.r1.channels = c1 # Each sink's type must be defined a1.sinks.k1.type = logger #Specify the channel the sink should use a1.sinks.k1.channel = c1 # Each channel's type is defined. a1.channels.c1.type = memory # Other config values specific to each type of channel(sink or source) # can be defined as well # In this case, it specifies the capacity of the memory channel a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 2.在flume的安装目录/flume-1.5.2下运行 cd /app/flume-1.5.2 ./bin/flume-ng agent --conf ./conf/ --conf-file ./conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console 3.再打开一个终端，输入如下命令： telnet localhost 44444 hello world 注：在CentOS6.5运行telnet提示"command not found"，使用sudo yum install telnet进行安装 4.在原来的终端上查看，可以收到来自于telnet发出的消息 3.2.2测试收集日志到HDFS 1.在$FLUME_HOME/conf目录下修改flume-conf.properties.template文件，复制并改名为flume-conf2.properties cd /app/flume-1.5.2/conf cp flume-conf.properties.template flume-conf2.properties sudo vi flume-conf2.properties a1.sources = r1 a1.sinks = k1 a1.channels = c1 a1.sources.r1.type = exec a1.sources.r1.channels = c1 a1.sources.r1.command = tail -F /app/hadoop-1.1.2/logs/hadoop-shiyanlou-namenode-b393a04554e1.log a1.sinks.k1.type = hdfs a1.sinks.k1.channel = c1 a1.sinks.k1.hdfs.path = hdfs://hadoop:9000/class12/out_flume a1.sinks.k1.hdfs.filePrefix = events- a1.sinks.k1.hdfs.round = true a1.sinks.k1.hdfs.roundValue = 10 a1.sinks.k1.hdfs.roundUnit = minute a1.sinks.k1.hdfs.rollSize = 4000000 a1.sinks.k1.hdfs.rollCount = 0 a1.sinks.k1.hdfs.writeFormat = Text a1.sinks.k1.hdfs.fileType = DataStream a1.sinks.k1.hdfs.batchSize = 10 a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 2.在flume的安装目录/flume-1.5.2下运行 cd /app/flume-1.5.2 ./bin/flume-ng agent --conf ./conf/ --conf-file ./conf/flume-conf2.properties --name a1 -Dflume.root.logger=INFO,console 3.不断收集hadoop-hadoop-namenode-hadoop1.log的数据写入HDFS中 4.查看hdfs中/class12/out_flume中的文件 hadoop fs -ls /class12/out_flume hadoop fs -cat /class12/out_flume/events-.1433921305493 本文转自shishanyuan博客园博客，原文链接： http://www.cnblogs.com/shishanyuan/p/4648284.html，如需转载请自行联系原作者

2017-11-14

Provisioning Services 7.6 入门到精通系列之二：基础架构环境

在阅读本博文之前强烈建议同学们提前了解Citrix XenDesktop和XenApp等产品，并对PVS相关产品概念架构有深入的理解。 1.1以下是在上一章节中规划的本次测试环境的配置清单（后续会根据高可用架构进行扩展） No. 主机名服务器角色操作系统内存 IP地址备注 1 CTXDC01 AD/DHCP/DNS Windows Server 2012 R2 2G 192.168.1.50 已部署 2 CTXLIC01 License Server Windows Server 2012 R2 2G 192.168.1.51 已部署 3 CTXDB01 MS SQL Windows Server 2012 R2 4G 192.168.1.52 已部署 4 CTXPVS01 PVS Windows Server 2012 R2 4G 192.168.1.53 1.2 准备好AD/DNS/DHCP、License Server、SQL Server和PVS Server共4台虚拟机后如下： 1.3根据以下博文完成AD/DNS的部署和配置 XenApp_XenDesktop_7.6实战篇之三：AD、DNS服务器规划及部署 http://stephen1991.blog.51cto.com/8959108/1659598 XenApp_XenDesktop_7.6实战篇之四：AD、DNS服务器高级配置 http://stephen1991.blog.51cto.com/8959108/1659609 XenApp_XenDesktop_7.6实战篇之五：组织单元、用户（组）、权限规划及配置 http://stephen1991.blog.51cto.com/8959108/1661048 1.4另外需额外创建配置以下域用户和组类别账户名称作用及成员备注用户 CTXAdmin Citrix管理员用户 SQLAdmin 数据库管理员用户 SQLServices SQL服务账号密码永不过期用户 PVSServices PVS Stream Service和Soap Service账户密码永不过期用户组 CTXAdmins PVS场管理员组（成员：CTXAdmin、PVSServices）具备PVS服务器本地管理员权限用户组 SQLAdmins SQL Server管理员组（成员：CTXAdmins、SQLServices、SQLAdmin）具备SQL服务器本地管理员权限总体原则: PVS管理员ctxadmin和服务账号pvsservices必须对SQL具备管理员权限； PVS管理员和服务账号必须为PVS的本地管理员组成员； SQL管理员和服务账号必须为SQL服务器的本地管理员组成员为PVS目标设备在AD创建计算机账户时还必须保证ctxadmin具备在AD创建计算机账户的权限，项目中如果允许可将ctxadmin加入到Domain Admins组中。 1.5根据以下博文完成DHCP的配置 XenApp_XenDesktop_7.6实战篇之六：DHCP服务器规划及部署 http://stephen1991.blog.51cto.com/8959108/1661053 1.6配置DHCP作用域选项66/67 1.7根据以下博文完成License Server的部署和配置 XenApp_XenDesktop_7.6实战篇之七：License Server规划及部署 http://stephen1991.blog.51cto.com/8959108/1661055 XenApp_XenDesktop_7.6实战篇之八：申请及导入许可证 http://stephen1991.blog.51cto.com/8959108/1661552 1.8根据以下博文完成SQL Server的部署和配置 XenApp_XenDesktop_7.6实战篇之九：SQL Server数据库服务器规划及部署 http://stephen1991.blog.51cto.com/8959108/1661577 在准备好AD/DNS/DHCP和数据库服务器，我们将在下一章节为大家分享PVS服务器的安装。本文转自 Stephen_huang 51CTO博客，原文链接：http://blog.51cto.com/stephen1991/1683324，如需转载请自行联系原作者

2017-11-14

Android GIS开发系列-- 入门季（4） GraphicsLayer的点击查询要素

上一讲中我们学会了如何在MapView中添加Graphic要素，那么在百度或高德地图中，当我们点击要素时，会显示出相应的详细信息。在GraphicsLayer中也提供了这样的方法。下面我们来学习在GraphicsLayer中如何点击查找要素。首先在MapView中添加两个Graphic要素。代码如下，注意这里用Graphic(Geometrygeometry,Symbolsymbol, Map<String,Object> attributes)来实例化Graphic，Map<String,Object> attributes是要素的属性值。详细见以下代码。TILED_WORLD_STREETS_URL为网络图层地址。 privatevoidinitLayer(){ mapView.addLayer(newArcGISTiledMapServiceLayer( TILED_WORLD_STREETS_URL)); graphicsLayer=newGraphicsLayer(); mapView.addLayer(graphicsLayer); Polygonpolygon=newPolygon(); polygon.startPath(newPoint(1.2575908509778766E7,2879410.9266042486)); polygon.lineTo(newPoint(1.284360696117901E7,3021972.232083669)); polygon.lineTo(newPoint(1.2826182801620414E7,2713089.403544925)); Map<String,Object>attr1=newHashMap<>(); attr1.put("name","广州"); attr1.put("mark","广州是南方的城市"); Graphicgraphic1=newGraphic(polygon,newSimpleFillSymbol(Color.RED),attr1); graphicsLayer.addGraphic(graphic1); Polygonpolygon2=newPolygon(); polygon2.startPath(newPoint(1.3388507951011453E7,3611225.628065273)); polygon2.lineTo(newPoint(1.3607101952746565E7,3858331.890896268)); polygon2.lineTo(newPoint(1.3613438010767872E7,3449656.14852193)); Map<String,Object>attr2=newHashMap<>(); attr2.put("name","上海"); attr2.put("mark","上海是中部的城市"); Graphicgraphic2=newGraphic(polygon2,newSimpleFillSymbol(Color.GREEN),attr2); graphicsLayer.addGraphic(graphic2); } 效果图如下：准备工作完成后，设置MapView的点击事件， mapView.setOnSingleTapListener(newOnSingleTapListener(){ @Override publicvoidonSingleTap(floatx,floaty){ //TODOAuto-generatedmethodstub handleSingleTap(x,y); } }); 在handleSingleTap方法中来处理查询事件，GraphicsLayer查询要用到getGraphicIDs(float x, float y, int tolerance, int numberOfResults)或者getGraphicIDs(float x, float y, int tolerance)方法，前面两个参数是地图点击时的 x与y的值，tolerance是围绕x与y这个点所查询的范围，numberOfResults是要返回结果的大小。 /** *GraphicsLayer的点击查询 *@paramx *@paramy */ protectedvoidhandleSingleTap(floatx,floaty){ int[]graphicIds=graphicsLayer.getGraphicIDs(x,y,8); if(graphicIds!=null&&graphicIds.length>0){ for(inti=0;i<graphicIds.length;i++){ Graphicgraphic=graphicsLayer.getGraphic(graphicIds[i]); Map<String,Object>attr=graphic.getAttributes(); Log.i(TAG,attr.get("name")+"===="+attr.get("mark")); } } }这样当我们点击要素时，会打出以下的信息。没有整理与归纳的知识，一文不值！高度概括与梳理的知识，才是自己真正的知识与技能。永远不要让自己的自由、好奇、充满创造力的想法被现实的框架所束缚，让创造力自由成长吧！多花时间，关心他（她）人，正如别人所关心你的。理想的腾飞与实现，没有别人的支持与帮助，是万万不能的。本文转自wenglabs博客园博客，原文链接：http://www.cnblogs.com/arxive/p/7751872.html ，如需转载请自行联系原作者

2017-11-13

Provisioning Services 7.6 入门到精通系列之一：PVS前期规划

鉴于PVS这么强大及在企业中应用非常广泛的产品，而且网上这方面的文档也很稀少，所以将撰写ProvisioningServices入门到精通系列博文，后期再通过51CTO博客制作PVS专题，这也即将是我在

2017-11-13

从入门到精通AD中的5种操作主机

在之前我们已经了解了在AD(活动目录)中创建林,域树和子域的方法,在一个域中,为了提高容错性和高可用性,我们建议大家在一个域中最好存在多台DC,每个DC维护域中相同的活动目录数据库.而这些DC是对等的,那么就会产生一些问题:为了保证活动目录数据库的一致性就需要执行复制操作,一般的复制是多主机复制(多个DC平等),但某些更改不适合使用多主机复制执行,因此需要有称为"操作主机"的DC接受此类更改的请求.首先我们先来了解什么是"操作主机","操作主机"都包括什么? 在每个林中有5种操作主机角色(这些操作主机角色可以指派给一个或多个DC) 在林范围内包括以下两种: 1)架构主机 2)域命名主机在每个林中这些角色都必须是唯一的! 在域范围内包括以下: 1)主域控制器仿真主机（PDC Emulator） 2)相对 ID (RID) 主机 3)基础结构主机以上三种在每个域中必须是唯一的! 1.架构主机（Schema Master）架构主机控制对整个林的架构的全部更新在整个林中，只能有一个架构主机如何管理架构主机(默认没有安装管理架构的工具): 1)注册架构管理工具regsvr32 schmmgmt.dll 2)使用mmc添加【Active Directory架构】 3)查看架构主机 2.域命名主机（Domain Naming Master）控制林中域的添加或删除，可以防止林中的域名重复在整个林中只能有一个域命名主机注意:任何运行WIN2003的DC都可以担当域命名主机这一角色,如果运行WIN2003的DC担当域命名主机角色,则必须启用为全局编录服务器使用【Active Directory域和信任关系】查看域命名主机 3.PDC 仿真主机（PDC Emulator Master） PDC 仿真主机作为混合模式域中的Windows NT PDC（主域控制器）林中的每个域中只能有一个PDC 仿真主机 PDC 仿真主机的主要作用: 1)管理来自客户端（Windows NT/95/98）的密码更改 2)最小化密码变化的复制等待时间 3)同步整个域内所有域控制器上的时间 4)查看PDC 仿真主机 4.RID 主机（RID Master） RID 主机将相对 ID（RID）序列分配给域中每个域控制器林中的每个域中只能有一个RID 主机每次当DC创建用户、组或计算机对象时，它就给该对象指派一个唯一的安全ID（SID）。SID包含一个域SID（它与域中创建的所有SID相同）和一个RID（它对域中创建的每个SID是唯一的）。对象的SID=域SID+RID 使用【Active Directory用户和计算机】查看RID主机 5.基础结构主机（Infrastructure Master）负责更新从它所在的域中的对象到其他域中对象的引用每个域中只能有一个基础结构主机基础结构主机将其数据与全局编录的数据进行比较,全局编录通过复制操作接受所有域中对象的定期更新,从而使全局编录的数据始终保持更新.如果基础结构主机发现数据已过时时,则它会从全局编录请求更新的数据,然后,基础结构主机再将这些更新的数据复制到域中的其他DC! 使用基础结构主机的要点: 1)除非域中只有一个域控制器，否则不应将基础结构主机角色指派给全局编录所在的域控制器 2)如果域中的所有域控制器都存有全局编录 ,则无论哪个域控制器承担基础结构主机角色均不重要 3)负责在重命名或更改组成员时更新“组到用户”的引用操作主机角色总结: 接下来我们来做转移操作主机角色案例:BENET公司组建了一个单域benet.com.cn,有两台DC，第一台DC的硬件配置比较低,第二台DC的硬件配置较高,目前的5个操作主机角色都在第一台DC上，如何将之转移到第二台DC上? 1)转移PDC主机,RID主机和基础结构主机角色 a)打开"AD用户与计算机"工具,右击"AD用户与计算机"然后单击"连接到域控制器",在"输入另一个DC的名称"窗口中,输入要担任PDC主机角色的DC的名称(dc2.benet.com.cn)或单击可用的DC列表中的该DC.在控制台树中,右击"AD用户与计算机",指向"所有任务",然后单击"操作主机"---"PDC"命令,显示当前的操作主机是"dc1.benet.com.cn": b)选择"更改": c)查看操作主机已经被转移成功,转移操作是可逆的! 2)转移架构主机角色 a)使用"AD架构"工具"来更改域控制器: b)指定名称: c)更改架构主机: 转移架构主机注意: 执行此过程的帐户必须是AD中Schema admins组的成员,后者必须被委派了相应的权限要想在控制台中添加AD架构管理单元,需要执行"regsvr32 schmmgmt.dll"命令注册该管理单元 3)转移域命名主机角色 a)打开"AD域和信任关系"工具,选择"连接到DC",在"输入另一个DC的名称"窗口中输入要担任域命名主机角色的DC的名称. b)在控制台中,右击"AD域和信任关系",然后单击"操作主机",显示域命名主机是"dc1.benet.com.cn" c)选择更改: 转移域命名主机角色要注意: 执行此过程的帐户必须是AD中的domain admins组或 enterprise admins组的成员,后被委派权限若要将角色转移到另一个DC,可能首先需要将"AD域和信任关系"的焦点改为目标DC,要执行此操作,右击"AD域和信任关系",在单击"连接到域控制器",然后单击目标DC即可! 当然还可以使用命令行方式转移主机角色以下显示转移RID主机的操作: 如果要转移其他角色,可以在fsmo maintenance命令提示符下,输入"help"命令: 命令意义 Transfer PDC 转移PDC仿真主机 Transfer PID master 转移RID主机 Transfer infrastructure master 转移基础结构主机 Transfer domain naming master 转移域命名主机 Transfer schema master 转移架构主机转移操作主机角色的总结: 在转移主机角色过程中，相关DC始终联机，因此没有数据损失在转移主机角色过程中要注意执行者是否有权限可以使用Windows图形界面和命令行方式实现转移操作主机的过程是可逆的刚才我们在做转移主机的时候,原来的DC还存在,属于联机状态下做的转移,那么如果操作主机角色所在的域控制器出了故障，并且无法恢复，如何解决?这时就不能通过转移的方法产生出新的操作主机角色,那么要通过占用操作主机的方法(或叫强制传送). 案例:BENET公司组建了一个单域,域名为benet.com.cn,有两台DC,目前的5个操作主机角色都在第一台DC上,某一天第一台DC出了故障,并且无法恢复,如何使第二台DC充当5个操作主机角色呢? 为了模拟第一台DC出故障,可以禁用第一台DC的网卡,使该计算机不能联网,这时第二台DC就联系不到操作主机,显示操作主机错误,如下图,角色不能被传送. 1）在“命令提示符”中键入“ntdsutil”命令 2）在ntdsutil命令提示符下键入“roles”命令 3）在fsmo maintenance命令提示符下,键入“connection” 命令 4）在server connections命令提示符下,键入 "connect to server dc2.benet.com.cn"(由于第一台DC脱机,需要使用第二台DC进行域名解析)命令 5）在server connections命令提示符下,键入“quit”（返回上一级命令提示符） 6）在fsmo maintenance命令提示符下,键入“seize RID master” ,按提示确认是否占用RID Master角色,单击"是"按钮,完成操作后,操作主机恢复为第二台DC: 最后查看操作主机: 如果要占用其他角色,可以在fsmo maintenance命令提示符下,键入"help"命令,使用以下命令来完成: 命令意义 Seize PDC 占用PDC仿真主机 Seize PID master 占用RID主机 Seize infrastructure master 占用基础结构主机 Seize domain naming master 占用域命名主机 Seize schema master 占用架构主机本文转自 149banzhang 51CTO博客，原文链接：http://blog.51cto.com/149banzhang/713303，如需转载请自行联系原作者

2017-11-13

Android GIS开发系列-- 入门季（3） GraphicsLayer添加点、线、面

GraphicsLayer是图形图层，可以自定义图形添加到地图上。调用GraphicsLayer的addGraphic方法就能添加图形，此方法要一个Graphic对象，此对象的构造方法是Graphic(Geometrygeometry,Symbolsymbol)，前面第一个是要素，而Geometry的子类有Envelope,MultiPath,MultiPoint,Point,Segment，Line,Polygon,Polyline。我们所常用的有点Point，线Polyline，面Polygon。而后面一个参数是样式，点的样式是SimpleMarkerSymbol，线的样式是SimpleLineSymbol，面的样式是SimpleFillSymbol。一、GraphicsLayer添加点： Graphicgraphic=newGraphic(point,newSimpleMarkerSymbol(Color.RED,5,SimpleMarkerSymbol.STYLE.CIRCLE)); graphicsLayer.addGraphic(graphic); 二、添加线，样式有实线、虚线、点虚线、点点虚线、点线等。效果图如下： Polylinepolyline=newPolyline(); polyline.startPath(newPoint(113,23));//第一个点用startPath,后面的点用lineTo方法 polyline.lineTo(newPoint(123,23)); graphic=newGraphic(polyline,newSimpleLineSymbol(Color.RED,3,SimpleLineSymbol.STYLE.SOLID)); graphicsLayer.addGraphic(graphic); 三、添加面Polygon，效果图如下： Polygonpolygon=newPolygon(); //第一个点startPath,后面的点用lineTo polygon.startPath(newPoint(118,23)); polygon.lineTo(newPoint(118,15)); polygon.lineTo(newPoint(113,13)); graphic=newGraphic(polygon,newSimpleFillSymbol(Color.GREEN,SimpleFillSymbol.STYLE.SOLID)); graphicsLayer.addGraphic(graphic); Code for all: packagecom.arcgis.test; importandroid.graphics.Color; importandroid.os.Bundle; importandroid.support.annotation.Nullable; importandroid.support.v7.app.AppCompatActivity; importandroid.view.View; importandroid.widget.AdapterView; importandroid.widget.Button; importandroid.widget.Spinner; importcom.esri.android.map.GraphicsLayer; importcom.esri.android.map.MapView; importcom.esri.android.map.ags.ArcGISTiledMapServiceLayer; importcom.esri.android.map.event.OnSingleTapListener; importcom.esri.core.geometry.Point; importcom.esri.core.geometry.Polygon; importcom.esri.core.geometry.Polyline; importcom.esri.core.map.Graphic; importcom.esri.core.symbol.SimpleFillSymbol; importcom.esri.core.symbol.SimpleLineSymbol; importcom.esri.core.symbol.SimpleMarkerSymbol; importjava.util.ArrayList; importjava.util.List; publicclassGraphicActivityextendsAppCompatActivity{ privateMapViewmMapView; privateSpinnergraphicTypeSpinner; privateButtonclernBtn; privateArcGISTiledMapServiceLayerarcGISTiledMapServiceLayer; privateGraphicsLayergraphicsLayer; privateStringmapServerUrl="http://services.arcgisonline.com/ArcGIS/rest/services/World_Street_Map/MapServer"; //点集合 privateList<Point>pointList=newArrayList<>(); privateGraphicgraphic; @Override protectedvoidonCreate(@NullableBundlesavedInstanceState){ super.onCreate(savedInstanceState); setContentView(R.layout.activity_graphic); initView(); addLayer(); } privatevoidinitView(){ mMapView=(MapView)findViewById(R.id.map_view); //mapview点击事件 mMapView.setOnSingleTapListener(newOnSingleTapListener(){ @Override publicvoidonSingleTap(floatx,floaty){ handleSingleTap(x,y); } }); graphicTypeSpinner=(Spinner)findViewById(R.id.spinner_type); graphicTypeSpinner.setOnItemSelectedListener(newAdapterView.OnItemSelectedListener(){ @Override publicvoidonItemSelected(AdapterView<?>parent,Viewview,intposition,longid){ pointList.removeAll(pointList); graphicsLayer.removeAll(); } @Override publicvoidonNothingSelected(AdapterView<?>parent){ } }); clernBtn=(Button)findViewById(R.id.clear_graphic); clernBtn.setOnClickListener(newView.OnClickListener(){ @Override publicvoidonClick(Viewv){ pointList.removeAll(pointList); graphicsLayer.removeAll(); } }); } privatevoidaddLayer(){ arcGISTiledMapServiceLayer=newArcGISTiledMapServiceLayer(mapServerUrl); mMapView.addLayer(arcGISTiledMapServiceLayer); graphicsLayer=newGraphicsLayer(); mMapView.addLayer(graphicsLayer); } privatevoidhandleSingleTap(floatx,floaty){ Pointpoint=mMapView.toMapPoint(x,y); graphicsLayer.removeAll(); pointList.add(point); Stringtype=graphicTypeSpinner.getSelectedItem().toString().trim(); switch(type){ case"点": graphic=newGraphic(point,newSimpleMarkerSymbol(Color.RED,5,SimpleMarkerSymbol.STYLE.CIRCLE)); graphicsLayer.addGraphic(graphic); break; case"线": Polylinepolyline=newPolyline(); if(pointList.size()>1){ for(inti=0;i<pointList.size();i++){ if(i==0){ polyline.startPath(pointList.get(i)); }else{ polyline.lineTo(pointList.get(i)); } } } graphic=newGraphic(polyline,newSimpleLineSymbol(Color.RED,3,SimpleLineSymbol.STYLE.SOLID)); graphicsLayer.addGraphic(graphic); break; case"面": Polygonpolygon=newPolygon(); for(inti=0;i<pointList.size();i++){ if(i==0){ polygon.startPath(pointList.get(i)); }else{ polygon.lineTo(pointList.get(i)); } } graphic=newGraphic(polygon,newSimpleFillSymbol(Color.GREEN,SimpleFillSymbol.STYLE.SOLID)); graphicsLayer.addGraphic(graphic); break; } } } 没有整理与归纳的知识，一文不值！高度概括与梳理的知识，才是自己真正的知识与技能。永远不要让自己的自由、好奇、充满创造力的想法被现实的框架所束缚，让创造力自由成长吧！多花时间，关心他（她）人，正如别人所关心你的。理想的腾飞与实现，没有别人的支持与帮助，是万万不能的。本文转自wenglabs博客园博客，原文链接：http://www.cnblogs.com/arxive/p/7751859.html ，如需转载请自行联系原作者

2017-11-12

Hadoop MapReduce编程 API入门系列之网页排序（二十八）

Map output bytes=247 Map output materialized bytes=275 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=275 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1439694848 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=119 File Output Format Counters Bytes Written=113 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=17 success. 17 2016-12-13 19:07:44,783 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:44,796 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:44,799 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:45,231 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:45,245 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:45,266 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local455505407_0021 2016-12-13 19:07:45,483 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:45,484 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local455505407_0021 2016-12-13 19:07:45,484 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:45,485 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:45,495 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:45,495 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local455505407_0021_m_000000_0 2016-12-13 19:07:45,500 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:45,559 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@223b788b 2016-12-13 19:07:45,565 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr20/part-r-00000:0+101 2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:45,598 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:45,600 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:45,608 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:45,609 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:45,609 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:45,609 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 255; bufvoid = 104857600 2016-12-13 19:07:45,610 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:45,625 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:45,631 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local455505407_0021_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:45,638 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr20/part-r-00000:0+101 2016-12-13 19:07:45,639 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local455505407_0021_m_000000_0' done. 2016-12-13 19:07:45,639 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local455505407_0021_m_000000_0 2016-12-13 19:07:45,639 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:45,640 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:45,641 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local455505407_0021_r_000000_0 2016-12-13 19:07:45,645 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:45,690 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@7d5e1574 2016-12-13 19:07:45,691 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2bd2b2f9 2016-12-13 19:07:45,703 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:45,704 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local455505407_0021_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:45,709 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#21 about to shuffle output of map attempt_local455505407_0021_m_000000_0 decomp: 279 len: 283 to MEMORY 2016-12-13 19:07:45,710 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 279 bytes from map-output for attempt_local455505407_0021_m_000000_0 2016-12-13 19:07:45,711 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 279, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->279 2016-12-13 19:07:45,712 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:45,714 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:45,715 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:45,729 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:45,730 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 275 bytes 2016-12-13 19:07:45,732 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 279 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:45,734 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 283 bytes from disk 2016-12-13 19:07:45,734 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:45,734 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:45,736 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 275 bytes 2016-12-13 19:07:45,737 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.21104025855884012 3___________ *********** new pageRank value is 0.3575070100293671 5___________ *********** new pageRank value is 0.4016854782039228 6___________ *********** new pageRank value is 0.1286040574733812 1___________ 2016-12-13 19:07:45,749 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local455505407_0021_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:45,753 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:45,753 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local455505407_0021_r_000000_0 is allowed to commit now 2016-12-13 19:07:45,762 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local455505407_0021_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr21/_temporary/0/task_local455505407_0021_r_000000 2016-12-13 19:07:45,764 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:45,764 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local455505407_0021_r_000000_0' done. 2016-12-13 19:07:45,764 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local455505407_0021_r_000000_0 2016-12-13 19:07:45,765 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:46,485 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local455505407_0021 running in uber mode : false 2016-12-13 19:07:46,486 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:46,487 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local455505407_0021 completed successfully 2016-12-13 19:07:46,498 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=35976 FILE: Number of bytes written=8181702 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=255 Map output materialized bytes=283 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=283 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1588592640 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=121 File Output Format Counters Bytes Written=111 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=15 success. 15 2016-12-13 19:07:46,508 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:46,516 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:46,519 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:46,868 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:46,879 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:46,896 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1152686492_0022 2016-12-13 19:07:47,037 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:47,037 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1152686492_0022 2016-12-13 19:07:47,037 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:47,039 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:47,045 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:47,045 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1152686492_0022_m_000000_0 2016-12-13 19:07:47,048 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:47,105 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@35f8fa7b 2016-12-13 19:07:47,109 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr21/part-r-00000:0+99 2016-12-13 19:07:47,131 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:47,133 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:47,139 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 248; bufvoid = 104857600 2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:47,155 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:47,160 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1152686492_0022_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr21/part-r-00000:0+99 2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1152686492_0022_m_000000_0' done. 2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1152686492_0022_m_000000_0 2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:47,166 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:47,167 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1152686492_0022_r_000000_0 2016-12-13 19:07:47,169 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:47,215 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@6df4957a 2016-12-13 19:07:47,215 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5481b4fb 2016-12-13 19:07:47,217 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:47,219 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1152686492_0022_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:47,224 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#22 about to shuffle output of map attempt_local1152686492_0022_m_000000_0 decomp: 272 len: 276 to MEMORY 2016-12-13 19:07:47,225 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 272 bytes from map-output for attempt_local1152686492_0022_m_000000_0 2016-12-13 19:07:47,225 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 272, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->272 2016-12-13 19:07:47,227 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:47,228 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:47,228 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:47,237 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:47,238 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 268 bytes 2016-12-13 19:07:47,240 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 272 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:47,241 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 276 bytes from disk 2016-12-13 19:07:47,241 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:47,242 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:47,243 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 268 bytes 2016-12-13 19:07:47,244 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.2082163282366672 2___________ *********** new pageRank value is 0.3525651625503612 4___________ *********** new pageRank value is 0.396037682951149 5___________ *********** new pageRank value is 0.12719210988750704 1___________ 2016-12-13 19:07:47,256 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1152686492_0022_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:47,260 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:47,261 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1152686492_0022_r_000000_0 is allowed to commit now 2016-12-13 19:07:47,270 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1152686492_0022_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr22/_temporary/0/task_local1152686492_0022_r_000000 2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1152686492_0022_r_000000_0' done. 2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1152686492_0022_r_000000_0 2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:48,038 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1152686492_0022 running in uber mode : false 2016-12-13 19:07:48,039 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:48,040 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1152686492_0022 completed successfully 2016-12-13 19:07:48,048 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=37782 FILE: Number of bytes written=8572508 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=248 Map output materialized bytes=276 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=276 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=13 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1588592640 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=119 File Output Format Counters Bytes Written=110 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=12 success. 12 2016-12-13 19:07:48,059 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:48,070 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:48,073 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:48,425 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:48,446 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:48,465 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1854065889_0023 2016-12-13 19:07:48,612 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:48,613 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:48,613 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1854065889_0023 2016-12-13 19:07:48,615 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:48,621 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:48,622 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1854065889_0023_m_000000_0 2016-12-13 19:07:48,624 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:48,668 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@135c5527 2016-12-13 19:07:48,672 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr22/part-r-00000:0+98 2016-12-13 19:07:48,709 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:48,711 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:48,720 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 247; bufvoid = 104857600 2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:48,738 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:48,742 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1854065889_0023_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr22/part-r-00000:0+98 2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1854065889_0023_m_000000_0' done. 2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1854065889_0023_m_000000_0 2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:48,747 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:48,748 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1854065889_0023_r_000000_0 2016-12-13 19:07:48,751 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:48,798 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@21287ab4 2016-12-13 19:07:48,798 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@688fd06d 2016-12-13 19:07:48,801 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:48,802 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1854065889_0023_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:48,807 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#23 about to shuffle output of map attempt_local1854065889_0023_m_000000_0 decomp: 271 len: 275 to MEMORY 2016-12-13 19:07:48,809 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 271 bytes from map-output for attempt_local1854065889_0023_m_000000_0 2016-12-13 19:07:48,811 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 271, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->271 2016-12-13 19:07:48,815 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:48,817 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:48,818 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:48,831 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:48,832 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 267 bytes 2016-12-13 19:07:48,834 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 271 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:48,836 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 275 bytes from disk 2016-12-13 19:07:48,836 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:48,836 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:48,837 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 267 bytes 2016-12-13 19:07:48,839 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.20581601525423832 2___________ *********** new pageRank value is 0.3483646014570123 4___________ *********** new pageRank value is 0.3912370348699975 4___________ *********** new pageRank value is 0.12599193950058354 1___________ 2016-12-13 19:07:48,854 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1854065889_0023_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:48,858 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:48,859 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1854065889_0023_r_000000_0 is allowed to commit now 2016-12-13 19:07:48,870 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1854065889_0023_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr23/_temporary/0/task_local1854065889_0023_r_000000 2016-12-13 19:07:48,879 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:48,880 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1854065889_0023_r_000000_0' done. 2016-12-13 19:07:48,882 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1854065889_0023_r_000000_0 2016-12-13 19:07:48,883 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:49,614 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1854065889_0023 running in uber mode : false 2016-12-13 19:07:49,615 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:49,617 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1854065889_0023 completed successfully 2016-12-13 19:07:49,637 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=39570 FILE: Number of bytes written=8963305 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=247 Map output materialized bytes=275 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=275 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=13 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1755316224 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=118 File Output Format Counters Bytes Written=112 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=11 success. 11 2016-12-13 19:07:49,648 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:49,656 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:49,659 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:50,057 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:50,073 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:50,093 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local2101720639_0024 2016-12-13 19:07:50,248 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:50,249 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local2101720639_0024 2016-12-13 19:07:50,251 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:50,252 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:50,262 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:50,263 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local2101720639_0024_m_000000_0 2016-12-13 19:07:50,267 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:50,324 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@63a803dc 2016-12-13 19:07:50,329 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr23/part-r-00000:0+100 2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:50,406 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:50,407 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:50,417 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 253; bufvoid = 104857600 2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:50,437 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:50,442 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local2101720639_0024_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:50,447 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr23/part-r-00000:0+100 2016-12-13 19:07:50,448 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local2101720639_0024_m_000000_0' done. 2016-12-13 19:07:50,448 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local2101720639_0024_m_000000_0 2016-12-13 19:07:50,448 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:50,450 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:50,450 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local2101720639_0024_r_000000_0 2016-12-13 19:07:50,454 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:50,523 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@7e37d95 2016-12-13 19:07:50,524 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6849c53e 2016-12-13 19:07:50,526 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:50,535 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local2101720639_0024_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:50,539 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#24 about to shuffle output of map attempt_local2101720639_0024_m_000000_0 decomp: 277 len: 281 to MEMORY 2016-12-13 19:07:50,541 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 277 bytes from map-output for attempt_local2101720639_0024_m_000000_0 2016-12-13 19:07:50,541 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 277, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->277 2016-12-13 19:07:50,542 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:50,544 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:50,545 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:50,559 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:50,559 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 273 bytes 2016-12-13 19:07:50,562 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 277 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:50,563 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 281 bytes from disk 2016-12-13 19:07:50,563 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:50,563 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:50,564 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 273 bytes 2016-12-13 19:07:50,566 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.20377573981974895 2___________ *********** new pageRank value is 0.3447941205905482 3___________ *********** new pageRank value is 0.3871564855262084 4___________ *********** new pageRank value is 0.12497180648305128 1___________ 2016-12-13 19:07:50,580 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local2101720639_0024_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:50,584 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:50,585 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local2101720639_0024_r_000000_0 is allowed to commit now 2016-12-13 19:07:50,595 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local2101720639_0024_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr24/_temporary/0/task_local2101720639_0024_r_000000 2016-12-13 19:07:50,597 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:50,597 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local2101720639_0024_r_000000_0' done. 2016-12-13 19:07:50,598 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local2101720639_0024_r_000000_0 2016-12-13 19:07:50,598 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:51,250 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local2101720639_0024 running in uber mode : false 2016-12-13 19:07:51,252 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:51,253 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local2101720639_0024 completed successfully 2016-12-13 19:07:51,263 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=41372 FILE: Number of bytes written=9354121 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=253 Map output materialized bytes=281 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=281 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1755316224 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=120 File Output Format Counters Bytes Written=112 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=10 success. 10 2016-12-13 19:07:51,279 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:51,292 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:51,295 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:51,881 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:51,893 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:51,913 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local756232932_0025 2016-12-13 19:07:52,079 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:52,080 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local756232932_0025 2016-12-13 19:07:52,080 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:52,081 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:52,087 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:52,087 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local756232932_0025_m_000000_0 2016-12-13 19:07:52,090 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:52,137 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@2bbddda 2016-12-13 19:07:52,141 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr24/part-r-00000:0+100 2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:52,166 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:52,167 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 251; bufvoid = 104857600 2016-12-13 19:07:52,176 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:52,193 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:52,198 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local756232932_0025_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:52,203 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr24/part-r-00000:0+100 2016-12-13 19:07:52,203 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local756232932_0025_m_000000_0' done. 2016-12-13 19:07:52,204 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local756232932_0025_m_000000_0 2016-12-13 19:07:52,204 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:52,205 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:52,205 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local756232932_0025_r_000000_0 2016-12-13 19:07:52,209 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:52,264 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@74f508b9 2016-12-13 19:07:52,264 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@30fe5ac 2016-12-13 19:07:52,266 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:52,268 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local756232932_0025_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:52,274 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#25 about to shuffle output of map attempt_local756232932_0025_m_000000_0 decomp: 275 len: 279 to MEMORY 2016-12-13 19:07:52,276 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 275 bytes from map-output for attempt_local756232932_0025_m_000000_0 2016-12-13 19:07:52,277 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 275, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->275 2016-12-13 19:07:52,278 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:52,280 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:52,280 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:52,294 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:52,295 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes 2016-12-13 19:07:52,298 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 275 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:52,299 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 279 bytes from disk 2016-12-13 19:07:52,299 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:52,300 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:52,301 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes 2016-12-13 19:07:52,303 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.20204150634863857 1___________ *********** new pageRank value is 0.34175921352732863 3___________ *********** new pageRank value is 0.38368802025726273 3___________ *********** new pageRank value is 0.12410468942339331 0___________ 2016-12-13 19:07:52,321 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local756232932_0025_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:52,326 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:52,327 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local756232932_0025_r_000000_0 is allowed to commit now 2016-12-13 19:07:52,339 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local756232932_0025_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr25/_temporary/0/task_local756232932_0025_r_000000 2016-12-13 19:07:52,343 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:52,344 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local756232932_0025_r_000000_0' done. 2016-12-13 19:07:52,344 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local756232932_0025_r_000000_0 2016-12-13 19:07:52,346 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:53,080 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local756232932_0025 running in uber mode : false 2016-12-13 19:07:53,081 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:53,082 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local756232932_0025 completed successfully 2016-12-13 19:07:53,092 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=43182 FILE: Number of bytes written=9742963 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=251 Map output materialized bytes=279 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=279 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1774190592 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=120 File Output Format Counters Bytes Written=114 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=7 success. 7 2016-12-13 19:07:53,109 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:53,116 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:53,119 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:53,539 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:53,553 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:53,574 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local525031809_0026 2016-12-13 19:07:53,728 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:53,729 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local525031809_0026 2016-12-13 19:07:53,729 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:53,730 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:53,736 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:53,737 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local525031809_0026_m_000000_0 2016-12-13 19:07:53,739 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:53,787 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4190fa1c 2016-12-13 19:07:53,793 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr25/part-r-00000:0+102 2016-12-13 19:07:53,825 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:53,825 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:53,826 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:53,826 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:53,826 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:53,827 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:53,835 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:53,836 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:53,836 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:53,837 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 258; bufvoid = 104857600 2016-12-13 19:07:53,837 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:53,859 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:53,864 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local525031809_0026_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:53,868 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr25/part-r-00000:0+102 2016-12-13 19:07:53,869 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local525031809_0026_m_000000_0' done. 2016-12-13 19:07:53,869 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local525031809_0026_m_000000_0 2016-12-13 19:07:53,869 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:53,870 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:53,871 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local525031809_0026_r_000000_0 2016-12-13 19:07:53,874 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:53,929 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4ab0342e 2016-12-13 19:07:53,930 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7631de50 2016-12-13 19:07:53,932 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:53,934 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local525031809_0026_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:53,940 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#26 about to shuffle output of map attempt_local525031809_0026_m_000000_0 decomp: 282 len: 286 to MEMORY 2016-12-13 19:07:53,942 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 282 bytes from map-output for attempt_local525031809_0026_m_000000_0 2016-12-13 19:07:53,942 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 282, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->282 2016-12-13 19:07:53,944 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:53,946 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:53,947 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:53,959 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:53,960 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes 2016-12-13 19:07:53,962 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 282 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:53,963 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 286 bytes from disk 2016-12-13 19:07:53,963 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:53,963 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:53,964 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes 2016-12-13 19:07:53,965 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.20056740860933667 1___________ *********** new pageRank value is 0.3391795418124502 2___________ *********** new pageRank value is 0.38073982450317145 2___________ *********** new pageRank value is 0.1233676401981714 0___________ 2016-12-13 19:07:53,977 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local525031809_0026_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:53,981 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:53,981 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local525031809_0026_r_000000_0 is allowed to commit now 2016-12-13 19:07:53,988 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local525031809_0026_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr26/_temporary/0/task_local525031809_0026_r_000000 2016-12-13 19:07:53,990 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:53,990 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local525031809_0026_r_000000_0' done. 2016-12-13 19:07:53,991 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local525031809_0026_r_000000_0 2016-12-13 19:07:53,991 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:54,729 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local525031809_0026 running in uber mode : false 2016-12-13 19:07:54,730 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:54,730 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local525031809_0026 completed successfully 2016-12-13 19:07:54,739 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=45006 FILE: Number of bytes written=10131824 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=258 Map output materialized bytes=286 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=286 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1774190592 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=122 File Output Format Counters Bytes Written=112 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=5 success. 5 2016-12-13 19:07:54,749 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:54,756 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:54,759 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:55,178 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:55,190 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:55,211 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local582978068_0027 2016-12-13 19:07:55,356 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:55,357 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local582978068_0027 2016-12-13 19:07:55,357 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:55,358 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:55,366 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:55,367 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local582978068_0027_m_000000_0 2016-12-13 19:07:55,371 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:55,428 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4b81e280 2016-12-13 19:07:55,432 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr26/part-r-00000:0+100 2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:55,461 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:55,469 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:55,470 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:55,470 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:55,470 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 251; bufvoid = 104857600 2016-12-13 19:07:55,471 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:55,486 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:55,493 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local582978068_0027_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:55,499 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr26/part-r-00000:0+100 2016-12-13 19:07:55,500 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local582978068_0027_m_000000_0' done. 2016-12-13 19:07:55,500 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local582978068_0027_m_000000_0 2016-12-13 19:07:55,501 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:55,502 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:55,502 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local582978068_0027_r_000000_0 2016-12-13 19:07:55,507 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:55,570 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@314779f4 2016-12-13 19:07:55,571 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4b979b72 2016-12-13 19:07:55,573 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:55,577 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local582978068_0027_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:55,587 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#27 about to shuffle output of map attempt_local582978068_0027_m_000000_0 decomp: 275 len: 279 to MEMORY 2016-12-13 19:07:55,589 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 275 bytes from map-output for attempt_local582978068_0027_m_000000_0 2016-12-13 19:07:55,590 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 275, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->275 2016-12-13 19:07:55,591 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:55,595 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:55,595 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:55,609 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:55,609 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes 2016-12-13 19:07:55,611 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 275 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:55,613 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 279 bytes from disk 2016-12-13 19:07:55,613 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:55,613 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:55,614 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes 2016-12-13 19:07:55,616 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.19931442541384786 1___________ *********** new pageRank value is 0.33698682115703876 2___________ *********** new pageRank value is 0.37823385762480544 2___________ *********** new pageRank value is 0.12274114865896807 0___________ 2016-12-13 19:07:55,636 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local582978068_0027_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:55,640 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:55,640 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local582978068_0027_r_000000_0 is allowed to commit now 2016-12-13 19:07:55,649 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local582978068_0027_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr27/_temporary/0/task_local582978068_0027_r_000000 2016-12-13 19:07:55,657 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:55,657 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local582978068_0027_r_000000_0' done. 2016-12-13 19:07:55,658 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local582978068_0027_r_000000_0 2016-12-13 19:07:55,658 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:56,357 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local582978068_0027 running in uber mode : false 2016-12-13 19:07:56,358 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:56,359 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local582978068_0027 completed successfully 2016-12-13 19:07:56,366 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=46826 FILE: Number of bytes written=10520671 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=251 Map output materialized bytes=279 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=279 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1889533952 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=120 File Output Format Counters Bytes Written=114 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=5 success. 5 2016-12-13 19:07:56,374 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:56,384 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:56,386 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:56,854 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:56,866 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:56,880 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1006109257_0028 2016-12-13 19:07:57,050 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:57,051 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1006109257_0028 2016-12-13 19:07:57,051 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:57,052 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:57,059 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:57,059 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1006109257_0028_m_000000_0 2016-12-13 19:07:57,062 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:57,114 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@753f2425 2016-12-13 19:07:57,118 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr27/part-r-00000:0+102 2016-12-13 19:07:57,152 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:57,152 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:57,152 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:57,153 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:57,153 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:57,154 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:57,162 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 258; bufvoid = 104857600 2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:57,175 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:57,179 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1006109257_0028_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:57,183 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr27/part-r-00000:0+102 2016-12-13 19:07:57,183 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1006109257_0028_m_000000_0' done. 2016-12-13 19:07:57,184 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1006109257_0028_m_000000_0 2016-12-13 19:07:57,184 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:57,185 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:57,185 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1006109257_0028_r_000000_0 2016-12-13 19:07:57,188 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:57,237 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@b901239 2016-12-13 19:07:57,237 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1f9c621d 2016-12-13 19:07:57,239 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:57,242 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1006109257_0028_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:57,246 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#28 about to shuffle output of map attempt_local1006109257_0028_m_000000_0 decomp: 282 len: 286 to MEMORY 2016-12-13 19:07:57,247 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 282 bytes from map-output for attempt_local1006109257_0028_m_000000_0 2016-12-13 19:07:57,247 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 282, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->282 2016-12-13 19:07:57,248 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:57,250 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:57,250 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:57,259 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:57,260 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes 2016-12-13 19:07:57,263 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 282 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:57,263 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 286 bytes from disk 2016-12-13 19:07:57,264 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:57,264 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:57,265 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes 2016-12-13 19:07:57,266 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.1982493894905423 1___________ *********** new pageRank value is 0.33512300847148907 1___________ *********** new pageRank value is 0.3761037861635444 2___________ *********** new pageRank value is 0.12220863080088534 0___________ 2016-12-13 19:07:57,280 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1006109257_0028_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:57,284 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:57,284 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1006109257_0028_r_000000_0 is allowed to commit now 2016-12-13 19:07:57,295 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1006109257_0028_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr28/_temporary/0/task_local1006109257_0028_r_000000 2016-12-13 19:07:57,299 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:57,299 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1006109257_0028_r_000000_0' done. 2016-12-13 19:07:57,299 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1006109257_0028_r_000000_0 2016-12-13 19:07:57,300 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:58,051 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1006109257_0028 running in uber mode : false 2016-12-13 19:07:58,052 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:58,053 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1006109257_0028 completed successfully 2016-12-13 19:07:58,057 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=48650 FILE: Number of bytes written=10911508 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=258 Map output materialized bytes=286 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=286 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1889533952 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=122 File Output Format Counters Bytes Written=112 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=4 success. 4 2016-12-13 19:07:58,066 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2016-12-13 19:07:58,073 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-12-13 19:07:58,076 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-12-13 19:07:58,528 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1 2016-12-13 19:07:58,540 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2016-12-13 19:07:58,559 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local626952993_0029 2016-12-13 19:07:58,716 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2016-12-13 19:07:58,717 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local626952993_0029 2016-12-13 19:07:58,717 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2016-12-13 19:07:58,718 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2016-12-13 19:07:58,725 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2016-12-13 19:07:58,726 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local626952993_0029_m_000000_0 2016-12-13 19:07:58,728 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:58,780 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3e0a19cb 2016-12-13 19:07:58,784 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr28/part-r-00000:0+100 2016-12-13 19:07:58,831 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584) 2016-12-13 19:07:58,831 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100 2016-12-13 19:07:58,831 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080 2016-12-13 19:07:58,832 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600 2016-12-13 19:07:58,832 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600 2016-12-13 19:07:58,833 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-12-13 19:07:58,839 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 2016-12-13 19:07:58,839 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output 2016-12-13 19:07:58,840 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output 2016-12-13 19:07:58,840 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 252; bufvoid = 104857600 2016-12-13 19:07:58,840 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600 2016-12-13 19:07:58,853 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0 2016-12-13 19:07:58,858 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local626952993_0029_m_000000_0 is done. And is in the process of committing 2016-12-13 19:07:58,863 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr28/part-r-00000:0+100 2016-12-13 19:07:58,863 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local626952993_0029_m_000000_0' done. 2016-12-13 19:07:58,864 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local626952993_0029_m_000000_0 2016-12-13 19:07:58,864 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2016-12-13 19:07:58,865 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks 2016-12-13 19:07:58,866 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local626952993_0029_r_000000_0 2016-12-13 19:07:58,870 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2016-12-13 19:07:58,926 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@6f36ac8a 2016-12-13 19:07:58,926 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6e79d110 2016-12-13 19:07:58,928 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2016-12-13 19:07:58,930 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local626952993_0029_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 2016-12-13 19:07:58,935 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#29 about to shuffle output of map attempt_local626952993_0029_m_000000_0 decomp: 276 len: 280 to MEMORY 2016-12-13 19:07:58,936 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 276 bytes from map-output for attempt_local626952993_0029_m_000000_0 2016-12-13 19:07:58,937 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 276, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->276 2016-12-13 19:07:58,938 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning 2016-12-13 19:07:58,940 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:58,940 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 2016-12-13 19:07:58,949 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:58,949 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 272 bytes 2016-12-13 19:07:58,952 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 276 bytes to disk to satisfy reduce memory limit 2016-12-13 19:07:58,953 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 280 bytes from disk 2016-12-13 19:07:58,953 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce 2016-12-13 19:07:58,954 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments 2016-12-13 19:07:58,954 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 272 bytes 2016-12-13 19:07:58,956 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. *********** new pageRank value is 0.19734410911950637 0___________ *********** new pageRank value is 0.33353876774336305 1___________ *********** new pageRank value is 0.37429322529114195 1___________ *********** new pageRank value is 0.12175599053348046 0___________ 2016-12-13 19:07:58,969 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local626952993_0029_r_000000_0 is done. And is in the process of committing 2016-12-13 19:07:58,978 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied. 2016-12-13 19:07:58,979 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local626952993_0029_r_000000_0 is allowed to commit now 2016-12-13 19:07:58,996 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local626952993_0029_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr29/_temporary/0/task_local626952993_0029_r_000000 2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce 2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local626952993_0029_r_000000_0' done. 2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local626952993_0029_r_000000_0 2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete. 2016-12-13 19:07:59,717 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local626952993_0029 running in uber mode : false 2016-12-13 19:07:59,718 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100% 2016-12-13 19:07:59,719 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local626952993_0029 completed successfully 2016-12-13 19:07:59,728 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34 File System Counters FILE: Number of bytes read=50472 FILE: Number of bytes written=11300358 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=4 Map output records=11 Map output bytes=252 Map output materialized bytes=280 Input split bytes=139 Combine input records=0 Combine output records=0 Reduce input groups=4 Reduce shuffle bytes=280 Reduce input records=11 Reduce output records=4 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=21 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=1901068288 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=120 File Output Format Counters Bytes Written=114 zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter my=2 success. 2 代码 package zhouls.bigdata.myMapReduce.pagerank; //pr值 import java.io.IOException; import java.util.Arrays; import org.apache.commons.lang.StringUtils; public class Node { private double pageRank=1.0; private String[] adjacentNodeNames; public static final char fieldSeparator = '\t';//分隔符 public double getPageRank() { return pageRank; } public Node setPageRank(double pageRank) { this.pageRank = pageRank; return this; } public String[] getAdjacentNodeNames() { return adjacentNodeNames; } public Node setAdjacentNodeNames(String[] adjacentNodeNames) { this.adjacentNodeNames = adjacentNodeNames; return this; }//这一大段的get和set，可以右键，source，产生get和set，自动生成。 public boolean containsAdjacentNodes() { return adjacentNodeNames != null && adjacentNodeNames.length>0; } @Override public String toString() { StringBuilder sb = new StringBuilder(); sb.append(pageRank); if (getAdjacentNodeNames() != null) { sb.append(fieldSeparator) .append(StringUtils .join(getAdjacentNodeNames(), fieldSeparator)); } return sb.toString(); } //value =1.0 B D public static Node fromMR(String value) throws IOException { String[] parts = StringUtils.splitPreserveAllTokens( value, fieldSeparator); if (parts.length < 1) { throw new IOException( "Expected 1 or more parts but received " + parts.length); } Node node = new Node() .setPageRank(Double.valueOf(parts[0])); if (parts.length > 1) { node.setAdjacentNodeNames(Arrays.copyOfRange(parts, 1, parts.length)); } return node; } } package zhouls.bigdata.myMapReduce.pagerank;//网页排名 import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class RunJob { public static enum Mycounter{ my } public static void main(String[] args) { Configuration config =new Configuration(); // config.set("fs.defaultFS", "hdfs://HadoopMaster:9000"); // config.set("yarn.resourcemanager.hostname", "HadoopMaster"); double d =0.001; int i=0; while(true){ i++; try { config.setInt("runCount", i); FileSystem fs =FileSystem.get(config); Job job =Job.getInstance(config); job.setJarByClass(RunJob.class); job.setJobName("pr"+i); job.setMapperClass(PageRankMapper.class); job.setReducerClass(PageRankReducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); job.setInputFormatClass(KeyValueTextInputFormat.class); // Path inputPath =new Path("hdfs://HadoopMaster:9000/pagerank/pagerank.txt"); Path inputPath =new Path("./data/pagerank/pagerank.txt"); if(i>1){ // inputPath =new Path("hdfs://HadoopMaster:9000/out/pagerank/pr"+(i-1)); inputPath =new Path("./out/pagerank/pr"+(i-1)); } FileInputFormat.addInputPath(job, inputPath); // Path outpath =new Path("hdfs://HadoopMaster:9000/out/pagerank/pr"+i); Path outpath =new Path("./out/pagerank/pr"+i); if(fs.exists(outpath)){ fs.delete(outpath, true); } FileOutputFormat.setOutputPath(job, outpath); boolean f= job.waitForCompletion(true); if(f){ System.out.println("success."); long sum= job.getCounters().findCounter(Mycounter.my).getValue(); System.out.println(sum); double avgd= sum/4000.0; if(avgd<d){ break; } } } catch (Exception e) { e.printStackTrace(); } } } static class PageRankMapper extends Mapper<Text, Text, Text, Text>{ protected void map(Text key, Text value, Context context) throws IOException, InterruptedException { int runCount= context.getConfiguration().getInt("runCount", 1); String page =key.toString(); Node node =null; if(runCount==1){ node =Node.fromMR("1.0"+"\t"+value.toString()); }else{ node =Node.fromMR(value.toString()); } context.write(new Text(page), new Text(node.toString()));//A:1.0 B D if(node.containsAdjacentNodes()){ double outValue =node.getPageRank()/node.getAdjacentNodeNames().length; for (int i = 0; i < node.getAdjacentNodeNames().length; i++) { String outPage = node.getAdjacentNodeNames()[i]; context.write(new Text(outPage), new Text(outValue+""));//B:0.5 D:0.5 } } } } static class PageRankReducer extends Reducer<Text, Text, Text, Text>{ protected void reduce(Text arg0, Iterable<Text> arg1, Context arg2) throws IOException, InterruptedException { double sum =0.0; Node sourceNode =null; for(Text i:arg1){ Node node =Node.fromMR(i.toString()); if(node.containsAdjacentNodes()){ sourceNode =node; }else{ sum=sum+node.getPageRank(); } } double newPR=(0.15/4)+(0.85*sum); System.out.println("*********** new pageRank value is "+newPR); //把新的pr值和计算之前的pr比较 double d= newPR -sourceNode.getPageRank(); int j=(int)( d*1000.0); j=Math.abs(j); System.out.println(j+"___________"); arg2.getCounter(Mycounter.my).increment(j);; sourceNode.setPageRank(newPR); arg2.write(arg0, new Text(sourceNode.toString())); } } } 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6171162.html，如需转载请自行联系原作者

2017-11-12

HBase编程 API入门系列之delete（客户端而言）（3）

hbase(main):001:0>scan 'test_table2' ROW COLUMN+CELL row_04 column=f1:name, timestamp=1478117286377, value=Andy3 row_04 column=f2:name, timestamp=1478117286377, value=Andy3 1 row(s) in 1.5670 seconds hbase(main):002:0> 如现在这样，不指定，默认把所有的列全会删除（开发实际中，一定要加指定呀） 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.util.Bytes; 13 14 public class HBaseTest { 15 public static void main(String[] args) throws Exception { 16 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 17 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 18 // put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 19 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 20 // table.put(put); 21 22 // Get get = new Get(Bytes.toBytes("row_04")); 23 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 24 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 25 // System.out.println(rest.toString()); 26 // table.close(); 27 28 Delete delete = new Delete(Bytes.toBytes("row_04")); 29 table.delete(delete); 30 table.close(); 31 } 32 33 public static Configuration getConfig(){ 34 Configuration configuration = new Configuration(); 35 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 36 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 37 return configuration; 38 } 39 } 2016-12-10 15:37:29,818 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x5fc2fc59 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 15:37:29,838 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 15:37:29,838 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 15:37:29,839 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 15:37:29,839 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 15:37:29,839 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 15:37:29,839 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 15:37:29,840 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 15:37:29,842 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 15:37:29,842 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 15:37:29,842 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 15:37:29,843 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 15:37:29,843 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 15:37:29,844 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 15:37:29,844 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 15:37:29,844 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 15:37:29,847 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x5fc2fc590x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 15:37:29,925 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopSlave1/192.168.80.11:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 15:37:29,971 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopSlave1/192.168.80.11:2181, initiating session 2016-12-10 15:37:29,996 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopSlave1/192.168.80.11:2181, sessionid = 0x2587338ecb4000e, negotiated timeout = 40000 2016-12-10 15:37:31,324 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x2587338ecb4000e 2016-12-10 15:37:31,340 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x2587338ecb4000e closed 2016-12-10 15:37:31,344 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down hbase(main):001:0>scan 'test_table2' ROW COLUMN+CELL row_04 column=f1:name, timestamp=1478117286377, value=Andy3 row_04 column=f2:name, timestamp=1478117286377, value=Andy3 1 row(s) in 1.5670 seconds hbase(main):002:0>scan 'test_table2' ROW COLUMN+CELL 0 row(s) in 0.1720 seconds hbase(main):003:0> hbase(main):003:0>put 'test_table2','row_1','f1:name','andy' 0 row(s) in 0.6390 seconds hbase(main):004:0>put 'test_table2','row_1','f1:age','25' 0 row(s) in 0.0630 seconds hbase(main):005:0>put 'test_table2','row_1','f1:email','10086@qq.com' 0 row(s) in 0.2040 seconds hbase(main):006:0>scan 'test_table2' ROW COLUMN+CELL row_1 column=f1:age, timestamp=1478119757136, value=25 row_1 column=f1:email, timestamp=1478119978163, value=10086@qq.com row_1 column=f1:name, timestamp=1478119459805, value=andy 1 row(s) in 0.1000 seconds hbase(main):007:0>put 'test_table2','row_2','f1:email','10086@qq.com' 0 row(s) in 0.0290 seconds hbase(main):008:0>put 'test_table2','row_2','f1:name','maizi' 0 row(s) in 0.0940 seconds hbase(main):009:0>scan 'test_table2' ROW COLUMN+CELL row_1 column=f1:age, timestamp=1478119757136, value=25 row_1 column=f1:email, timestamp=1478119978163, value=10086@qq.com row_1 column=f1:name, timestamp=1478119459805, value=andy row_2 column=f1:email, timestamp=1478120078440, value=10086@qq.com row_2 column=f1:name, timestamp=1478120133536, value=maizi 2 row(s) in 0.1280 seconds hbase(main):010:0> 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.util.Bytes; 13 14 public class HBaseTest { 15 public static void main(String[] args) throws Exception { 16 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 17 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 18 // put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 19 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 20 // table.put(put); 21 22 // Get get = new Get(Bytes.toBytes("row_04")); 23 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 24 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 25 // System.out.println(rest.toString()); 26 // table.close(); 27 28 Delete delete = new Delete(Bytes.toBytes("row_1")); 29 delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); 30 table.delete(delete); 31 table.close(); 32 } 33 34 public static Configuration getConfig(){ 35 Configuration configuration = new Configuration(); 36 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 37 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 38 return configuration; 39 } 40 } 2016-12-10 16:01:36,462 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x27758c30 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 16:01:36,493 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 16:01:36,493 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 16:01:36,493 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 16:01:36,494 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 16:01:36,494 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 16:01:36,494 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 16:01:36,495 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 16:01:36,496 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 16:01:36,496 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 16:01:36,496 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 16:01:36,496 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 16:01:36,496 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 16:01:36,496 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 16:01:36,497 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 16:01:36,497 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 16:01:36,500 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x27758c300x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 16:01:36,664 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 16:01:36,679 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 16:01:36,764 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf4000f, negotiated timeout = 40000 2016-12-10 16:01:38,454 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf4000f 2016-12-10 16:01:38,461 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf4000f closed 2016-12-10 16:01:38,461 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down hbase(main):015:0>scan 'test_table2' ROW COLUMN+CELL row_1 column=f1:age, timestamp=1478119757136, value=25 row_1 column=f1:email, timestamp=1478119978163, value=10086@qq.com row_1 column=f1:name, timestamp=1478119459805, value=andy row_2 column=f1:email, timestamp=1478120078440, value=10086@qq.com row_2 column=f1:name, timestamp=1478120133536, value=maizi 2 row(s) in 0.3150 seconds hbase(main):016:0>scan 'test_table2' ROW COLUMN+CELL row_1 column=f1:age, timestamp=1478119757136, value=25 row_1 column=f1:name, timestamp=1478119459805, value=andy row_2 column=f1:email, timestamp=1478120078440, value=10086@qq.com row_2 column=f1:name, timestamp=1478120133536, value=maizi 2 row(s) in 0.1080 seconds hbase(main):017:0> 若是删除不存在的列修饰符，看下会是什么情况 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.util.Bytes; 13 14 public class HBaseTest { 15 public static void main(String[] args) throws Exception { 16 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 17 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 18 // put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 19 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 20 // table.put(put); 21 22 // Get get = new Get(Bytes.toBytes("row_04")); 23 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 24 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 25 // System.out.println(rest.toString()); 26 // table.close(); 27 28 Delete delete = new Delete(Bytes.toBytes("row_1")); 29 delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email----"));//若是删除不存在的列修饰符，看下会是什么情况 30 table.delete(delete); 31 table.close(); 32 } 33 34 public static Configuration getConfig(){ 35 Configuration configuration = new Configuration(); 36 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 37 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 38 return configuration; 39 } 40 } 2016-12-10 16:10:59,651 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x39ce5ac9 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 16:10:59,684 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 16:10:59,684 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 16:10:59,684 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 16:10:59,685 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 16:10:59,685 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 16:10:59,685 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 16:10:59,687 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 16:10:59,688 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 16:10:59,688 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 16:10:59,688 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 16:10:59,689 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 16:10:59,689 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 16:10:59,689 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 16:10:59,690 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 16:10:59,690 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 16:10:59,696 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x39ce5ac90x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 16:10:59,928 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 16:11:00,063 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 16:11:00,154 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf40012, negotiated timeout = 40000 2016-12-10 16:11:02,018 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf40012 2016-12-10 16:11:02,029 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf40012 closed 2016-12-10 16:11:02,029 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 若是删除不存在的列族，则会出现什么情况？ 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.util.Bytes; 13 14 public class HBaseTest { 15 public static void main(String[] args) throws Exception { 16 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 17 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 18 // put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 19 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 20 // table.put(put); 21 22 // Get get = new Get(Bytes.toBytes("row_04")); 23 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 24 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 25 // System.out.println(rest.toString()); 26 // table.close(); 27 28 Delete delete = new Delete(Bytes.toBytes("row_1")); 29 delete.deleteColumn(Bytes.toBytes("f3"), Bytes.toBytes("email"));//若是删除不存在的列族，则会出现什么情况？ 30 table.delete(delete); 31 table.close(); 32 } 33 34 public static Configuration getConfig(){ 35 Configuration configuration = new Configuration(); 36 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 37 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 38 return configuration; 39 } 40 } 2016-12-10 16:13:54,520 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x1646b78d connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 16:13:54,560 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 16:13:54,561 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 16:13:54,561 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 16:13:54,561 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 16:13:54,564 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 16:13:54,566 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 16:13:54,568 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 16:13:54,569 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 16:13:54,569 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 16:13:54,570 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 16:13:54,570 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 16:13:54,570 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 16:13:54,571 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 16:13:54,571 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 16:13:54,572 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 16:13:54,576 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x1646b78d0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 16:13:54,768 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 16:13:54,778 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 16:13:54,855 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf40013, negotiated timeout = 40000 Exception in thread "main" org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family f3 does not exist in region test_table2,,1478116411036.086c41857c60fbce5f79f61c45269784. in table 'test_table2', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'f2', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3562) at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:2604) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2283) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33646) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:329) at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:978) at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:965) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136) at org.apache.hadoop.hbase.client.HTable.delete(HTable.java:982) at zhouls.bigdata.HbaseProject.Test1.HBaseTest.main(HBaseTest.java:31) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException): org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family f3 does not exist in region test_table2,,1478116411036.086c41857c60fbce5f79f61c45269784. in table 'test_table2', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'f2', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:3562) at org.apache.hadoop.hbase.regionserver.HRegion.delete(HRegion.java:2604) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2283) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33646) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2178) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.mutate(ClientProtos.java:34082) at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:975) ... 4 more 这也是，证明，为什么HBase在创建表时，列簇是必须要，列可不要？ 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.util.Bytes; 13 14 public class HBaseTest { 15 public static void main(String[] args) throws Exception { 16 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 17 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 18 // put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 19 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 20 // table.put(put); 21 22 // Get get = new Get(Bytes.toBytes("row_04")); 23 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 24 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 25 // System.out.println(rest.toString()); 26 // table.close(); 27 28 Delete delete = new Delete(Bytes.toBytes("row_2")); 29 delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); 30 delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); 31 table.delete(delete); 32 table.close(); 33 } 34 35 public static Configuration getConfig(){ 36 Configuration configuration = new Configuration(); 37 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 38 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 39 return configuration; 40 } 41 } 2016-12-10 16:20:39,421 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x27758c30 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 16:20:39,445 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 16:20:39,446 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 16:20:39,446 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 16:20:39,446 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 16:20:39,446 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 16:20:39,447 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 16:20:39,448 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 16:20:39,448 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 16:20:39,448 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 16:20:39,449 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 16:20:39,449 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 16:20:39,449 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 16:20:39,449 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 16:20:39,450 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 16:20:39,450 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 16:20:39,452 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x27758c300x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 16:20:39,562 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopSlave1/192.168.80.11:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 16:20:39,567 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopSlave1/192.168.80.11:2181, initiating session 2016-12-10 16:20:39,595 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopSlave1/192.168.80.11:2181, sessionid = 0x2587338ecb40011, negotiated timeout = 40000 2016-12-10 16:20:41,278 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x2587338ecb40011 2016-12-10 16:20:41,283 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x2587338ecb40011 closed 2016-12-10 16:20:41,284 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down hbase(main):017:0>scan 'test_table2' ROW COLUMN+CELL row_1 column=f1:age, timestamp=1478119757136, value=25 row_1 column=f1:name, timestamp=1478119459805, value=andy row_2 column=f1:email, timestamp=1478120078440, value=10086@qq.com row_2 column=f1:name, timestamp=1478120133536, value=maizi 2 row(s) in 0.1520 seconds hbase(main):018:0>scan 'test_table2' ROW COLUMN+CELL row_1 column=f1:age, timestamp=1478119757136, value=25 row_1 column=f1:name, timestamp=1478119459805, value=andy 1 row(s) in 0.1110 seconds hbase(main):019:0> 这就是，若不指定删除的列，则 Delete delete = new Delete(Bytes.toBytes("row_2")); 会删除，全部的列。这就是，若指定删除的列，则 delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); 只删除，被指定的列本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6155978.html，如需转载请自行联系原作者

2017-11-12

HBase编程 API入门系列之get（客户端而言）（2）

1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Get; 9 import org.apache.hadoop.hbase.client.HTable; 10 import org.apache.hadoop.hbase.client.Put; 11 import org.apache.hadoop.hbase.util.Bytes; 12 13 public class HBaseTest { 14 public static void main(String[] args) throws Exception { 15 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 16 // Put put = new Put(Bytes.toBytes("row_03"));//行键是row_01 17 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f，列修饰符是name，值是Andy3 18 // table.put(put); 19 20 Get get = new Get(Bytes.toBytes("row_03")); 21 org.apache.hadoop.hbase.client.Result rest = table.get(get); 22 System.out.println(rest.toString()); 23 table.close(); 24 } 25 26 public static Configuration getConfig(){ 27 Configuration configuration = new Configuration(); 28 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 29 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 30 return configuration; 31 } 32 } 2016-12-10 14:37:25,665 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x62ba5f6a connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 14:37:25,687 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 14:37:25,687 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 14:37:25,687 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 14:37:25,688 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 14:37:25,688 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 14:37:25,688 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 14:37:25,689 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 14:37:25,690 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 14:37:25,690 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 14:37:25,691 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 14:37:25,691 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 14:37:25,691 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 14:37:25,692 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 14:37:25,692 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 14:37:25,693 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 14:37:25,696 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x62ba5f6a0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 14:37:25,810 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 14:37:25,819 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 14:37:25,905 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf40002, negotiated timeout = 40000 keyvalues={row_03/f:name/1478104823358/Put/vlen=5/seqid=0} 2016-12-10 14:37:27,208 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf40002 2016-12-10 14:37:27,214 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf40002 closed 2016-12-10 14:37:27,215 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Get; 9 import org.apache.hadoop.hbase.client.HTable; 10 import org.apache.hadoop.hbase.client.Put; 11 import org.apache.hadoop.hbase.util.Bytes; 12 13 public class HBaseTest { 14 public static void main(String[] args) throws Exception { 15 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 16 // Put put = new Put(Bytes.toBytes("row_03"));//行键是row_01 17 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f，列修饰符是name，值是Andy3 18 // table.put(put); 19 20 Get get = new Get(Bytes.toBytes("row_01")); 21 org.apache.hadoop.hbase.client.Result rest = table.get(get); 22 System.out.println(rest.toString()); 23 table.close(); 24 } 25 26 public static Configuration getConfig(){ 27 Configuration configuration = new Configuration(); 28 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 29 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 30 return configuration; 31 } 32 } 2016-12-10 14:40:43,222 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x114b320d connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 14:40:43,256 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 14:40:43,256 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 14:40:43,256 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 14:40:43,256 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 14:40:43,257 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 14:40:43,257 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 14:40:43,258 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 14:40:43,259 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 14:40:43,259 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 14:40:43,259 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 14:40:43,260 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 14:40:43,260 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 14:40:43,260 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 14:40:43,260 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 14:40:43,261 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 14:40:43,266 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x114b320d0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 14:40:43,392 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopSlave1/192.168.80.11:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 14:40:43,402 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopSlave1/192.168.80.11:2181, initiating session 2016-12-10 14:40:43,432 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopSlave1/192.168.80.11:2181, sessionid = 0x2587338ecb40004, negotiated timeout = 40000 keyvalues={row_01/f:col/1478102698687/Put/vlen=5/seqid=0, row_01/f:name/1478104345828/Put/vlen=4/seqid=0} 2016-12-10 14:40:45,146 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x2587338ecb40004 2016-12-10 14:40:45,153 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x2587338ecb40004 closed 2016-12-10 14:40:45,153 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Get; 9 import org.apache.hadoop.hbase.client.HTable; 10 import org.apache.hadoop.hbase.client.Put; 11 import org.apache.hadoop.hbase.util.Bytes; 12 13 public class HBaseTest { 14 public static void main(String[] args) throws Exception { 15 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 16 // Put put = new Put(Bytes.toBytes("row_03"));//行键是row_01 17 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f，列修饰符是name，值是Andy3 18 // table.put(put); 19 20 Get get = new Get(Bytes.toBytes("row_01")); 21 get.addColumn(Bytes.toBytes("f"), Bytes.toBytes("name")); 22 org.apache.hadoop.hbase.client.Result rest = table.get(get); 23 System.out.println(rest.toString()); 24 table.close(); 25 } 26 27 public static Configuration getConfig(){ 28 Configuration configuration = new Configuration(); 29 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 30 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 31 return configuration; 32 } 33 } 2016-12-10 14:43:35,385 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x62ba5f6a connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 14:43:35,408 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 14:43:35,409 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 14:43:35,409 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 14:43:35,409 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 14:43:35,409 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 14:43:35,410 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 14:43:35,411 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 14:43:35,412 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 14:43:35,412 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 14:43:35,413 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 14:43:35,413 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 14:43:35,413 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 14:43:35,413 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 14:43:35,414 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 14:43:35,414 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 14:43:35,416 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x62ba5f6a0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 14:43:35,518 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 14:43:35,525 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 14:43:35,590 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf40003, negotiated timeout = 40000 keyvalues={row_01/f:name/1478104345828/Put/vlen=4/seqid=0} 2016-12-10 14:43:37,380 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf40003 2016-12-10 14:43:37,386 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf40003 closed 2016-12-10 14:43:37,386 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Get; 9 import org.apache.hadoop.hbase.client.HTable; 10 import org.apache.hadoop.hbase.client.Put; 11 import org.apache.hadoop.hbase.util.Bytes; 12 13 public class HBaseTest { 14 public static void main(String[] args) throws Exception { 15 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 16 // Put put = new Put(Bytes.toBytes("row_03"));//行键是row_01 17 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f，列修饰符是name，值是Andy3 18 // table.put(put); 19 20 Get get = new Get(Bytes.toBytes("row_01")); 21 get.addColumn(Bytes.toBytes("f"), Bytes.toBytes("col")); 22 org.apache.hadoop.hbase.client.Result rest = table.get(get); 23 System.out.println(rest.toString()); 24 table.close(); 25 } 26 27 public static Configuration getConfig(){ 28 Configuration configuration = new Configuration(); 29 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 30 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 31 return configuration; 32 } 33 } 2016-12-10 14:45:42,225 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x39ce5ac9 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 14:45:42,264 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 14:45:42,265 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 14:45:42,266 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 14:45:42,266 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 14:45:42,267 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 14:45:42,267 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 14:45:42,269 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 14:45:42,276 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 14:45:42,276 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 14:45:42,276 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 14:45:42,277 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 14:45:42,277 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 14:45:42,277 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 14:45:42,277 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 14:45:42,278 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 14:45:42,281 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x39ce5ac90x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 14:45:42,444 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopSlave1/192.168.80.11:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 14:45:42,451 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopSlave1/192.168.80.11:2181, initiating session 2016-12-10 14:45:42,478 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopSlave1/192.168.80.11:2181, sessionid = 0x2587338ecb40005, negotiated timeout = 40000 keyvalues={row_01/f:col/1478102698687/Put/vlen=5/seqid=0} 2016-12-10 14:45:43,893 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x2587338ecb40005 2016-12-10 14:45:43,900 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x2587338ecb40005 closed 2016-12-10 14:45:43,900 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 在get里，有很多方法，大家自行去研究。大同小异。这里，为了更好的带领大家研究get，我再新建一个表。 create 'test_table2','f1','f2' 这是HBase里新建表的语句，test_table2是表名，f1和f2是列簇、 hbase(main):001:0>create 'test_table2','f1','f2' 0 row(s) in 15.5960 seconds => Hbase::Table - test_table2 hbase(main):002:0>desc 'test_table2' Table test_table2 is ENABLED test_table2 COLUMN FAMILIES DESCRIPTION {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} {NAME => 'f2', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2 row(s) in 1.7530 seconds hbase(main):003:0> 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Get; 9 import org.apache.hadoop.hbase.client.HTable; 10 import org.apache.hadoop.hbase.client.Put; 11 import org.apache.hadoop.hbase.util.Bytes; 12 13 public class HBaseTest { 14 public static void main(String[] args) throws Exception { 15 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 16 Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 17 put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 18 put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 19 table.put(put); 20 // 21 // Get get = new Get(Bytes.toBytes("row_01")); 22 // get.addColumn(Bytes.toBytes("f"), Bytes.toBytes("col")); 23 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 24 // System.out.println(rest.toString()); 25 table.close(); 26 } 27 28 public static Configuration getConfig(){ 29 Configuration configuration = new Configuration(); 30 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 31 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 32 return configuration; 33 } 34 } hbase(main):001:0>scan 'test_table2' ROW COLUMN+CELL row_04 column=f1:name, timestamp=1478117286377, value=Andy3 row_04 column=f2:name, timestamp=1478117286377, value=Andy3 1 row(s) in 1.5670 seconds hbase(main):002:0> 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Get; 9 import org.apache.hadoop.hbase.client.HTable; 10 import org.apache.hadoop.hbase.client.Put; 11 import org.apache.hadoop.hbase.util.Bytes; 12 13 public class HBaseTest { 14 public static void main(String[] args) throws Exception { 15 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 16 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 17 // put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 18 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 19 // table.put(put); 20 21 Get get = new Get(Bytes.toBytes("row_04")); 22 get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));//比如去拿一个不存在的列修饰符，看会得到什么？ 23 org.apache.hadoop.hbase.client.Result rest = table.get(get); 24 System.out.println(rest.toString()); 25 table.close(); 26 } 27 28 public static Configuration getConfig(){ 29 Configuration configuration = new Configuration(); 30 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 31 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 32 return configuration; 33 } 34 } 2016-12-10 15:13:01,074 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x39ce5ac9 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 15:13:01,104 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 15:13:01,105 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 15:13:01,105 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 15:13:01,105 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 15:13:01,106 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 15:13:01,106 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 15:13:01,109 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 15:13:01,110 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 15:13:01,111 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 15:13:01,111 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 15:13:01,112 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 15:13:01,112 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 15:13:01,113 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 15:13:01,113 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 15:13:01,114 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 15:13:01,118 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x39ce5ac90x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 15:13:01,249 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 15:13:01,255 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 15:13:01,326 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf4000d, negotiated timeout = 40000 keyvalues=NONE 2016-12-10 15:13:03,155 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf4000d 2016-12-10 15:13:03,163 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf4000d closed 2016-12-10 15:13:03,163 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 如现在这样，不指定，默认把所有的列全拿出来（开发实际中，一定要加指定呀） 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Get; 9 import org.apache.hadoop.hbase.client.HTable; 10 import org.apache.hadoop.hbase.client.Put; 11 import org.apache.hadoop.hbase.util.Bytes; 12 13 public class HBaseTest { 14 public static void main(String[] args) throws Exception { 15 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table 16 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_01 17 // put.add(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f1，列修饰符是name，值是Andy3 18 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 19 // table.put(put); 20 21 Get get = new Get(Bytes.toBytes("row_04")); 22 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 23 org.apache.hadoop.hbase.client.Result rest = table.get(get); 24 System.out.println(rest.toString()); 25 table.close(); 26 } 27 28 public static Configuration getConfig(){ 29 Configuration configuration = new Configuration(); 30 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 31 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 32 return configuration; 33 } 34 } 2016-12-10 15:17:07,672 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x7a45682c connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 15:17:07,694 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 15:17:07,695 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 15:17:07,695 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 15:17:07,696 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 15:17:07,696 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 15:17:07,696 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 15:17:07,698 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 15:17:07,698 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 15:17:07,698 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 15:17:07,698 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 15:17:07,699 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 15:17:07,699 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 15:17:07,699 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 15:17:07,699 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 15:17:07,700 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 15:17:07,702 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x7a45682c0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 15:17:07,819 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 15:17:07,823 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 15:17:07,870 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf4000e, negotiated timeout = 40000 keyvalues={row_04/f1:name/1478117286377/Put/vlen=5/seqid=0, row_04/f2:name/1478117286377/Put/vlen=5/seqid=0} 2016-12-10 15:17:09,275 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf4000e 2016-12-10 15:17:09,283 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf4000e closed 2016-12-10 15:17:09,283 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6155713.html，如需转载请自行联系原作者

2017-11-12

HBase编程 API入门系列之put（客户端而言）（1）

[hadoop@HadoopSlave1 conf]$ cat regionservers HadoopMaster HadoopSlave1 HadoopSlave2 <configuration> <property> <name>hbase.zookeeper.quorum</name> <value>HadoopMaster,HadoopSlave1,HadoopSlave2</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://HadoopMaster:9000/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.tmp.dir</name> <value>/home/hadoop/data/hbase-1.2.3/tmp</value> </property> <property> <name>zookeeper.znode.parent</name> <value>/hbase</value> </property> </configuration> export JAVA_HOME=/home/hadoop/app/jdk1.7.0_79 export HBASE_MANAGES_ZK=false （外装的Zookeeper集群） hbase(main):002:0> create 'test_table','f' 语法： create table name , column family test_table 是HBase的数据表名，f是列簇为了后续的编程，我这里直接先手动创建最简单的hbase数据表。在代码里可以修改。 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import org.apache.hadoop.conf.Configuration; 4 import org.apache.hadoop.hbase.HBaseConfiguration; 5 import org.apache.hadoop.hbase.TableName; 6 import org.apache.hadoop.hbase.client.HTable; 7 import org.apache.hadoop.hbase.client.Put; 8 import org.apache.hadoop.hbase.util.Bytes; 9 10 11 public class HBaseTest { 12 public static void main(String[] args) throws Exception { 13 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 14 Put put = new Put(Bytes.toBytes("row_01"));//行键是row_01 15 put.add(Bytes.toBytes("f"),Bytes.toBytes("col"),Bytes.toBytes("maizi"));//列簇是f，列修饰符是col，值是maizi 16 table.put(put); 17 table.close(); 18 } 19 20 public static Configuration getConfig(){ 21 Configuration configuration = new Configuration(); 22 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 23 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 24 return configuration; 25 } 26 } 2016-12-10 11:05:45,077 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x5fc2fc59 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 11:05:45,115 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 11:05:45,115 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 11:05:45,115 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 11:05:45,115 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 11:05:45,116 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 11:05:45,116 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 11:05:45,118 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 11:05:45,119 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 11:05:45,120 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 11:05:45,120 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 11:05:45,121 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 11:05:45,121 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 11:05:45,131 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 11:05:45,131 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 11:05:45,132 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 11:05:45,136 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x5fc2fc590x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 11:05:45,329 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 11:05:45,365 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 11:05:45,421 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x1582587a9550008, negotiated timeout = 40000 2016-12-10 11:05:47,266 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x1582587a9550008 2016-12-10 11:05:47,275 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1582587a9550008 closed 2016-12-10 11:05:47,275 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down scan 'test_table' 全盘扫描该Hbase数据表test_table 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import org.apache.hadoop.conf.Configuration; 4 import org.apache.hadoop.hbase.HBaseConfiguration; 5 import org.apache.hadoop.hbase.TableName; 6 import org.apache.hadoop.hbase.client.HTable; 7 import org.apache.hadoop.hbase.client.Put; 8 import org.apache.hadoop.hbase.util.Bytes; 9 10 11 public class HBaseTest { 12 public static void main(String[] args) throws Exception { 13 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 14 Put put = new Put(Bytes.toBytes("row_01"));//行键是row_01 15 put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy"));//列簇是f，列修饰符是col，值是maizi 16 table.put(put); 17 table.close(); 18 } 19 20 public static Configuration getConfig(){ 21 Configuration configuration = new Configuration(); 22 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 23 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 24 return configuration; 25 } 26 } scan 'test_table' 全盘扫描该Hbase数据表test_table 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import org.apache.hadoop.conf.Configuration; 4 import org.apache.hadoop.hbase.HBaseConfiguration; 5 import org.apache.hadoop.hbase.TableName; 6 import org.apache.hadoop.hbase.client.HTable; 7 import org.apache.hadoop.hbase.client.Put; 8 import org.apache.hadoop.hbase.util.Bytes; 9 10 11 public class HBaseTest { 12 public static void main(String[] args) throws Exception { 13 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 14 Put put = new Put(Bytes.toBytes("row_02"));//行键是row_01 15 put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy2"));//列簇是f，列修饰符是col，值是maizi 16 table.put(put); 17 table.close(); 18 } 19 20 public static Configuration getConfig(){ 21 Configuration configuration = new Configuration(); 22 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 23 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 24 return configuration; 25 } 26 } scan 'test_table' 全盘扫描该Hbase数据表test_table 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import org.apache.hadoop.conf.Configuration; 4 import org.apache.hadoop.hbase.HBaseConfiguration; 5 import org.apache.hadoop.hbase.TableName; 6 import org.apache.hadoop.hbase.client.HTable; 7 import org.apache.hadoop.hbase.client.Put; 8 import org.apache.hadoop.hbase.util.Bytes; 9 10 11 public class HBaseTest { 12 public static void main(String[] args) throws Exception { 13 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 14 Put put = new Put(Bytes.toBytes("row_03"));//行键是row_01 15 put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f，列修饰符是col，值是maizi 16 table.put(put); 17 table.close(); 18 } 19 20 public static Configuration getConfig(){ 21 Configuration configuration = new Configuration(); 22 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 23 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 24 return configuration; 25 } 26 } scan 'test_table' 全盘扫描该Hbase数据表test_table 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6155220.html，如需转载请自行联系原作者

2017-11-12

OpenStack入门修炼之如何创建生产使用的openstack镜像（16）

在linux-node1节点上：、 [root@linux-node1 ~]# yum install -y openstack-nova-compute [root@linux-node1 ~]# yum install -y libvirt qemu-kvm [root@linux-node1 ~]# systemctl start libvirtd [root@linux-node1 ~]# systemctl enable libvirtd [root@linux-node1 ~]# vim /etc/nova/nova.conf 增加novncproxy的地址： novncproxy_base_url=http://192.168.56.11:6080/vnc_auto.html [root@linux-node1 ~]# systemctl start openstack-nova-compute [root@linux-node1 ~]# systemctl enable openstack-nova-compute [root@linux-node1 ~]# source admin-openstack [root@linux-node1 ~]# nova service-list 当需要删除该节点时，只能从命令行进行删除:nova service-delete ID。在horizon界面上是只能关闭服务，无法进行删除。创建openstack镜像的过程：（1）创建一个虚拟机硬盘，格式qcow2，大小为10G [root@linux-node1 ~]# qemu-img create -f qcow2 /tmp/centos.qcow2 10G Formatting '/tmp/centos.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 （2）创建虚拟机 [root@linux-node1 ~]# virt-install --virt-type kvm --name centos --ram 1024 \ --disk /tmp/centos.qcow2,format=qcow2 \ --network network=default \ --graphics vnc,listen=0.0.0.0 --noautoconsole \ --os-type=linux --os-variant=centos7.0 \ --location=/tmp/CentOS-7-x86_64-Minimal-1511.iso [root@linux-node1 ~]# virsh list --all Id 名称状态 ---------------------------------------------------- - centos 关闭 [root@linux-node1 ~]# virsh start centos （3）TightVNC链接192.168.56.11进入系统安装只分根分区，私有云不分swap分区（4）启动虚拟机 [root@linux-node1 ~]# virsh start centos （5）在虚拟机内安装默认的软件，或基础优化 yum install net-tools tree screen wget git vim （6）openstack上传镜像 [root@linux-node1 ~]# openstack image create "Centos-7-x86_64" --file=/tmp/centos.qcow2 --disk-format qcow2 --container-format bare --public +------------------+------------------------------------------------------+ | Field | Value | +------------------+------------------------------------------------------+ | checksum | 4ac6fb14f7e5bb3c9889df4bda71474a | | container_format | bare | | created_at | 2017-12-07T08:06:55Z | | disk_format | qcow2 | | file | /v2/images/5aa6e9a1-4aea-467c-a684-51080c326887/file | | id | 5aa6e9a1-4aea-467c-a684-51080c326887 | | min_disk | 0 | | min_ram | 0 | | name | Centos-7-x86_64 | | owner | 21a45d59913d4c05b46ad3ec92e61656 | | protected | False | | schema | /v2/schemas/image | | size | 1145896960 | | status | active | | tags | | | updated_at | 2017-12-07T08:08:42Z | | virtual_size | None | | visibility | public | +------------------+------------------------------------------------------+ [root@linux-node1 ~]# openstack image list +--------------------------------------+-----------------+--------+ | ID | Name | Status | +--------------------------------------+-----------------+--------+ | 5aa6e9a1-4aea-467c-a684-51080c326887 | Centos-7-x86_64 | active | | 51e1e125-dbe3-49c6-a3d5-55c89f195f55 | cirros | active | +--------------------------------------+-----------------+--------+ （7）通过上传的镜像创建centos7云主机使用admin账户创建云主机类型，后使用demo用户进行创建云主机本文转自 IT_外卖小哥 51CTO博客，原文链接:http://blog.51cto.com/jinlong/2049779

2017-11-11

OpenStack入门修炼之KVM虚拟机介绍和管理（4）

1.查看虚拟机，启动虚拟机 [root@linux-node1 ~]# virsh list --all Id Name State ---------------------------------------------------- - CentOS-7-x86_64 shut off [root@linux-node1 ~]# virsh start CentOS-7-x86_64 通过TightVNC连接，修改网络vi /etc/sysconfig/network-scripts/ifcfg-eth0 tips: KVM虚拟机在宿主机上的体现是一个进程，可以通过ps axu |grep kvm进行查看。对KVM的管理，就是在宿主机上对KVM的进程进行管理。 2.libvirt介绍 libvirt组成部分：libvirt daemon后台进程，libvirt API库，User-space management tools tips：（1）如果libvirtd服务停用了，会影响命令行管理虚拟机，但不会影响虚拟机本身的运行。（2）kvm是为openstack进行创建虚拟机，如果openstack服务停用，可以使用libvirtd服务命令行进行管理。 3.KVM常用管理：CPU,内存，网络（1）libvirt通过对xml文件进行管理 [root@linux-node1 ~]# cd /etc/libvirt/qemu [root@linux-node1 qemu]# ls -l total 8 -rw-------. 1 root root 3855 Nov 27 10:00 CentOS-7-x86_64.xml drwx------. 3 root root 4096 Nov 27 09:33 networks （2）CentOS-7-x86_64.xml是自动生成的，无法进行修改。如果需要修改，需要通过virsh edit进行编辑。但是在openstack上是无法进行修改的。 [root@linux-node1 qemu]# virsh edit CentOS-7-x86_64 （3）KVM修改热添加:virsh edit CentOS-7-x86_64，热添加会影响系统的性能，不适合在生产环境使用 tips：KVM支持热添加，公有云是不支持热添加的。需要宿主机和虚拟机的系统保持一致性。 ①cpu的热添加，将cpu核数改为2,openstack不支持修改为： [root@linux-node1 qemu]# virsh setvcpus CentOS-7-x86_64 2 --live ②内存热添加 a.查看KVM内存： [root@linux-node1 ~]# virsh qemu-monitor-command CentOS-7-x86_64 --hmp --cmd info balloon balloon: actual=1024 b.修改，将内存调成512M： [root@linux-node1 ~]# virsh qemu-monitor-command CentOS-7-x86_64 --hmp --cmd balloon 512 c.增加内存：修改xml文件，修改完xml文件需要重启虚拟机 <memory unit='KiB'>1048576</memory> 修改为： <memory unit='KiB'>1548576</memory> <currentMemory unit='KiB'>1048576</currentMemory> [root@linux-node1 ~]# virsh shutdown CentOS-7-x86_64 [root@linux-node1 ~]# virsh start CentOS-7-x86_64 （4）网络管理，配置KVM的桥接网络 [root@linux-node1 ~]# brctl show bridge name bridge id STP enabled interfaces virbr0 8000.52540067fb26 yes virbr0-nic vnet ①创建桥接网卡，写成脚本 [root@linux-node1 ~]# vim bridge.sh #!/bin/bash brctl addbr br0 brctl addif br0 ens33 ip addr del dev ens33 192.168.56.11/24 ifconfig br0 192.168.56.11/24 up route add default gw 192.168.56.2 [root@linux-node1 ~]# brctl show bridge name bridge id STP enabled interfaces br0 8000.000c2903f3fe no eth0 virbr0 8000.52540067fb26 yes virbr0-nic vnet0 ②修改xml [root@linux-node1 ~]# virsh edit CentOS-7-x86_64 interface type='network' 修改为 interface type='bridge' source network='default'/ 修改为 source bridge='br0'/ ③重启KVM虚拟机 [root@linux-node1 ~]# virsh shutdown CentOS-7-x86_64 Domain CentOS-7-x86_64 is being shutdown [root@linux-node1 ~]# virsh start CentOS-7-x86_64 Domain CentOS-7-x86_64 started ④修改虚拟机网卡配置 [root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0 TYPE=Ethernet BOOTPROTO=static NAME=eth0 ONBOOT=yes IPADDR=192.168.56.111 NETMASK=255.255.255.0 GATEWAY=192.168.56.2 [root@localhost ~]# systemctl restart network ⑤验证宿主机上能否登陆 [root@linux-node1 ~]# ping 192.168.56.111 PING 192.168.56.111 (192.168.56.111) 56(84) bytes of data. 64 bytes from 192.168.56.111: icmp_seq=1 ttl=64 time=0.618 ms 64 bytes from 192.168.56.111: icmp_seq=2 ttl=64 time=0.387 ms 64 bytes from 192.168.56.111: icmp_seq=3 ttl=64 time=0.264 ms 64 bytes from 192.168.56.111: icmp_seq=4 ttl=64 time=0.328 ms ^C --- 192.168.56.111 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3002ms rtt min/avg/max/mdev = 0.264/0.399/0.618/0.134 ms [root@linux-node1 ~]# ssh 192.168.56.111 The authenticity of host '192.168.56.111 (192.168.56.111)' can't be established. ECDSA key fingerprint is SHA256:k6ODnQ4pdtSMtYrfti2wB8eB5/XfCENW0SWApJrHTgY. ECDSA key fingerprint is MD5:06:9f:34:32:b8:b2:a3:91:12:f9:3b:8c:dd:c0:e7:63. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.56.111' (ECDSA) to the list of known hosts. root@192.168.56.111's password: Last login: Mon Nov 27 11:44:22 2017 版权声明：原创作品，谢绝转载。否则将追究法律责任本文转自 IT_外卖小哥 51CTO博客，原文链接:http://blog.51cto.com/jinlong/2049418

2017-11-11

OpenStack入门修炼之KVM性能优化及IO缓存介绍（5）

1.KVM的性能优化，介绍CPU,内存，IO性能优化 KVM CPU-->qemu进行模拟 ring 3-->用户应用（用户态，用户空间） ring 0-->操作系统（内核态，内核空间）虚拟化难题？（1）CPU优化：上下文切换：宿主机 ring 0 ring 3 虚拟机 ring 0 ring 3 处理器设置-->Inter VT-x/EPT或AMD-V/RVI（V）缓存： L1-->静态内存 L2、L3-->动态内存虚拟机在宿主机上是一个进程，受进程调度器的管理。容易导致Cache miss，为了减少缓存miss，可以通过taskset进行设定，把某个进程绑定到CPU的某个核上，进行提高性能。openstack上是无法进行绑定。 taskset -cp 1 25718 （2）内存优化：空间优化寻址物理在使用内存的时候：虚拟内存-->映射-->物理内存，使用KVM的时候会有如下内存使用过程虚拟机：虚拟内存 ↓映射虚拟机：物理内存 ↓映射宿主机：虚拟内存 ↓映射宿主机：物理内存影子页表技术加速寻址技术： Inter的EPT技术，内存优化技术，可以加速虚拟机到物理内存的映射 cat /proc/meminfo/ 默认内存是4k，内存是分页的。大页内存 [root@linux-node1 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never ksmd内存合并，扫描内存地址，把相同的内容进行合并释放，减少内存的占用 [root@linux-node1 ~]# ps axu |grep ksm root 27 0.0 0.0 0 0 ? SN 09:06 0:00 [ksmd] （3）I/O优化： ①通过virtio技术，加速io访问，linux默认就有这个功能，windows需要装载驱动 ②IO调度算法内核2.6有4种调度算法：Noop CFQ AS Deadline 默认CFQ 内核3.10有3中调度算法：Noop CFQ Deadline 默认Deadline 查看默认的IO调度算法 [root@linux-node1 ~]# dmesg |grep -i scheduler [ 0.300671] io scheduler noop registered [ 0.300674] io scheduler deadline registered (default) [ 0.300691] io scheduler cfq registered Noop算法：按顺序来，最简单的的I/O调度算法。该算法仅适当合并用户请求，并不排序请求：新的请求通常被插在调度队列的开头或末尾，下一个要处理的请求总是队列中的第一个请求。SSD必须设置Noop，性能最优。 2.KVM的IO缓存介绍服务器的硬盘缓存是关闭的，因为会使用RAID卡缓存，RAID带电池，不会在断电的时候丢失数据。虚拟机的应用程序写数据的方式选择： KVM默认的方式是WriteThrough，最安全的，但是性能也最差。本文转自 IT_外卖小哥 51CTO博客，原文链接:http://blog.51cto.com/jinlong/2049435

2017-11-11

HBase编程 API入门系列之scan（客户端而言）（5）

心得，写在前面的话，也许，中间会要多次执行，连接超时，多试试就好了。 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.client.ResultScanner; 13 import org.apache.hadoop.hbase.client.Scan; 14 import org.apache.hadoop.hbase.util.Bytes; 15 16 public class HBaseTest { 17 public static void main(String[] args) throws Exception { 18 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 19 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_04 20 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy1"));//列簇是f，列修饰符是name，值是Andy0 21 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 22 // table.put(put); 23 // table.close(); 24 25 // Get get = new Get(Bytes.toBytes("row_04")); 26 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 27 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 28 // System.out.println(rest.toString()); 29 // table.close(); 30 31 // Delete delete = new Delete(Bytes.toBytes("row_2")); 32 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); 33 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); 34 // table.delete(delete); 35 // table.close(); 36 37 // Delete delete = new Delete(Bytes.toBytes("row_04")); 38 //// delete.deleteColumn(Bytes.toBytes("f"), Bytes.toBytes("name"));//deleteColumn是删除某一个列簇里的最新时间戳版本。 39 // delete.deleteColumns(Bytes.toBytes("f"), Bytes.toBytes("name"));//delete.deleteColumns是删除某个列簇里的所有时间戳版本。 40 // table.delete(delete); 41 // table.close(); 42 43 44 Scan scan = new Scan(); 45 ResultScanner rst = table.getScanner(scan); 46 System.out.println(rst.toString()); 47 table.close(); 48 } 49 50 public static Configuration getConfig(){ 51 Configuration configuration = new Configuration(); 52 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 53 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 54 return configuration; 55 } 56 } 2016-12-10 17:24:08,394 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x7a45682c connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 17:24:08,432 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 17:24:08,433 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 17:24:08,434 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 17:24:08,435 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 17:24:08,436 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 17:24:08,436 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 17:24:08,439 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 17:24:08,442 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 17:24:08,443 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 17:24:08,443 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 17:24:08,443 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 17:24:08,444 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 17:24:08,444 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 17:24:08,445 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 17:24:08,445 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 17:24:08,449 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x7a45682c0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 17:24:08,638 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 17:24:08,651 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 17:24:08,688 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf4002b, negotiated timeout = 40000 org.apache.hadoop.hbase.client.ClientScanner@23b804dd 2016-12-10 17:24:10,158 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf4002b 2016-12-10 17:24:10,166 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf4002b closed 2016-12-10 17:24:10,167 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.client.ResultScanner; 13 import org.apache.hadoop.hbase.client.Scan; 14 import org.apache.hadoop.hbase.util.Bytes; 15 16 public class HBaseTest { 17 public static void main(String[] args) throws Exception { 18 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 19 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_04 20 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy1"));//列簇是f，列修饰符是name，值是Andy0 21 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 22 // table.put(put); 23 // table.close(); 24 25 // Get get = new Get(Bytes.toBytes("row_04")); 26 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 27 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 28 // System.out.println(rest.toString()); 29 // table.close(); 30 31 // Delete delete = new Delete(Bytes.toBytes("row_2")); 32 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); 33 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); 34 // table.delete(delete); 35 // table.close(); 36 37 // Delete delete = new Delete(Bytes.toBytes("row_04")); 38 //// delete.deleteColumn(Bytes.toBytes("f"), Bytes.toBytes("name"));//deleteColumn是删除某一个列簇里的最新时间戳版本。 39 // delete.deleteColumns(Bytes.toBytes("f"), Bytes.toBytes("name"));//delete.deleteColumns是删除某个列簇里的所有时间戳版本。 40 // table.delete(delete); 41 // table.close(); 42 43 44 Scan scan = new Scan(); 45 ResultScanner rst = table.getScanner(scan); 46 System.out.println(rst.toString()); 47 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 48 System.out.println(next.toString()); 49 } 50 table.close(); 51 52 } 53 54 public static Configuration getConfig(){ 55 Configuration configuration = new Configuration(); 56 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 57 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 58 return configuration; 59 } 60 } 2016-12-10 17:29:53,501 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x7a45682c connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 17:29:53,520 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 17:29:53,520 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 17:29:53,520 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 17:29:53,520 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 17:29:53,520 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 17:29:53,521 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 17:29:53,522 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 17:29:53,522 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 17:29:53,522 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 17:29:53,522 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 17:29:53,522 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 17:29:53,523 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 17:29:53,523 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 17:29:53,523 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 17:29:53,523 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 17:29:53,525 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x7a45682c0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 17:29:53,588 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopSlave1/192.168.80.11:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 17:29:53,594 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopSlave1/192.168.80.11:2181, initiating session 2016-12-10 17:29:53,612 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopSlave1/192.168.80.11:2181, sessionid = 0x2587338ecb40026, negotiated timeout = 40000 org.apache.hadoop.hbase.client.ClientScanner@ff35374 keyvalues={row_01/f:col/1478102698687/Put/vlen=5/seqid=0, row_01/f:name/1478104345828/Put/vlen=4/seqid=0} keyvalues={row_02/f:name/1478104477628/Put/vlen=5/seqid=0} keyvalues={row_03/f:name/1478123664884/Put/vlen=5/seqid=0} 2016-12-10 17:29:54,863 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x2587338ecb40026 2016-12-10 17:29:54,868 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x2587338ecb40026 closed 2016-12-10 17:29:54,868 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.HBaseConfiguration; 7 import org.apache.hadoop.hbase.TableName; 8 import org.apache.hadoop.hbase.client.Delete; 9 import org.apache.hadoop.hbase.client.Get; 10 import org.apache.hadoop.hbase.client.HTable; 11 import org.apache.hadoop.hbase.client.Put; 12 import org.apache.hadoop.hbase.client.ResultScanner; 13 import org.apache.hadoop.hbase.client.Scan; 14 import org.apache.hadoop.hbase.util.Bytes; 15 16 public class HBaseTest { 17 public static void main(String[] args) throws Exception { 18 HTable table = new HTable(getConfig(),TableName.valueOf("test_table2"));//表名是test_table2 19 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_04 20 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy1"));//列簇是f，列修饰符是name，值是Andy0 21 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 22 // table.put(put); 23 // table.close(); 24 25 // Get get = new Get(Bytes.toBytes("row_04")); 26 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 27 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 28 // System.out.println(rest.toString()); 29 // table.close(); 30 31 // Delete delete = new Delete(Bytes.toBytes("row_2")); 32 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); 33 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); 34 // table.delete(delete); 35 // table.close(); 36 37 // Delete delete = new Delete(Bytes.toBytes("row_04")); 38 //// delete.deleteColumn(Bytes.toBytes("f"), Bytes.toBytes("name"));//deleteColumn是删除某一个列簇里的最新时间戳版本。 39 // delete.deleteColumns(Bytes.toBytes("f"), Bytes.toBytes("name"));//delete.deleteColumns是删除某个列簇里的所有时间戳版本。 40 // table.delete(delete); 41 // table.close(); 42 43 44 Scan scan = new Scan(); 45 ResultScanner rst = table.getScanner(scan); 46 System.out.println(rst.toString()); 47 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 48 System.out.println(next.toString()); 49 } 50 table.close(); 51 } 52 53 public static Configuration getConfig(){ 54 Configuration configuration = new Configuration(); 55 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 56 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 57 return configuration; 58 } 59 } 2016-12-10 17:32:25,591 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x39ce5ac9 connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 17:32:25,612 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 17:32:25,613 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 17:32:25,613 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 17:32:25,613 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 17:32:25,613 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 17:32:25,613 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 17:32:25,615 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 17:32:25,615 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 17:32:25,615 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 17:32:25,616 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 17:32:25,616 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 17:32:25,616 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 17:32:25,616 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 17:32:25,616 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 17:32:25,617 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 17:32:25,619 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x39ce5ac90x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 17:32:25,701 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 17:32:25,705 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 17:32:25,762 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf4002e, negotiated timeout = 40000 org.apache.hadoop.hbase.client.ClientScanner@529d8124 keyvalues={row_1/f1:age/1478119757136/Put/vlen=2/seqid=0, row_1/f1:name/1478119459805/Put/vlen=4/seqid=0} 2016-12-10 17:32:27,091 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf4002e 2016-12-10 17:32:27,099 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf4002e closed 2016-12-10 17:32:27,100 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 1 Scan scan = new Scan(); 2 scan.setStartRow(Bytes.toBytes("3")); 3 ResultScanner rst = table.getScanner(scan); 4 System.out.println(rst.toString()); 5 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 6 System.out.println(next.toString()); 7 } 8 table.close(); 1 Scan scan = new Scan(); 2 scan.setStartRow(Bytes.toBytes("3"));//包含开始行键 3 scan.setStopRow(Bytes.toBytes("5"));//不包含结束行键 4 ResultScanner rst = table.getScanner(scan); 5 System.out.println(rst.toString()); 6 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 7 System.out.println(next.toString()); 8 } 9 table.close(); 1 Scan scan = new Scan(); 2 scan.setStartRow(Bytes.toBytes("1"));//包含开始行键 3 scan.setStopRow(Bytes.toBytes("5"));//不包含结束行键 4 ResultScanner rst = table.getScanner(scan); 5 System.out.println(rst.toString()); 6 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 7 System.out.println(next.toString()); 8 } 9 table.close(); 1 Scan scan = new Scan(); 2 scan.setStartRow(Bytes.toBytes("1"));//包含开始行键 3 scan.setStopRow(Bytes.toBytes("5"));//不包含结束行键 4 ResultScanner rst = table.getScanner(scan); 5 System.out.println(rst.toString()); 6 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 7 for(Cell cell:next.rawCells()){ 8 System.out.println(next.toString()); 9 System.out.println("family:" + Bytes.toString(CellUtil.cloneFamily(cell))); 10 System.out.println("col:" + Bytes.toString(CellUtil.cloneQualifier(cell))); 11 System.out.println("value" + Bytes.toString(CellUtil.cloneValue(cell))); 12 } 13 } 14 table.close(); 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.Cell; 7 import org.apache.hadoop.hbase.CellUtil; 8 import org.apache.hadoop.hbase.HBaseConfiguration; 9 import org.apache.hadoop.hbase.TableName; 10 import org.apache.hadoop.hbase.client.Delete; 11 import org.apache.hadoop.hbase.client.Get; 12 import org.apache.hadoop.hbase.client.HTable; 13 import org.apache.hadoop.hbase.client.Put; 14 import org.apache.hadoop.hbase.client.ResultScanner; 15 import org.apache.hadoop.hbase.client.Scan; 16 import org.apache.hadoop.hbase.util.Bytes; 17 18 public class HBaseTest { 19 public static void main(String[] args) throws Exception { 20 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 21 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_04 22 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy1"));//列簇是f，列修饰符是name，值是Andy0 23 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 24 // table.put(put); 25 // table.close(); 26 27 // Get get = new Get(Bytes.toBytes("row_04")); 28 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 29 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 30 // System.out.println(rest.toString()); 31 // table.close(); 32 33 // Delete delete = new Delete(Bytes.toBytes("row_2")); 34 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); 35 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); 36 // table.delete(delete); 37 // table.close(); 38 39 // Delete delete = new Delete(Bytes.toBytes("row_04")); 40 //// delete.deleteColumn(Bytes.toBytes("f"), Bytes.toBytes("name"));//deleteColumn是删除某一个列簇里的最新时间戳版本。 41 // delete.deleteColumns(Bytes.toBytes("f"), Bytes.toBytes("name"));//delete.deleteColumns是删除某个列簇里的所有时间戳版本。 42 // table.delete(delete); 43 // table.close(); 44 45 46 Scan scan = new Scan(); 47 scan.setStartRow(Bytes.toBytes("row_01"));//包含开始行键 48 scan.setStopRow(Bytes.toBytes("row_03"));//不包含结束行键 49 ResultScanner rst = table.getScanner(scan); 50 System.out.println(rst.toString()); 51 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 52 for(Cell cell:next.rawCells()){ 53 System.out.println(next.toString()); 54 System.out.println("family:" + Bytes.toString(CellUtil.cloneFamily(cell))); 55 System.out.println("col:" + Bytes.toString(CellUtil.cloneQualifier(cell))); 56 System.out.println("value" + Bytes.toString(CellUtil.cloneValue(cell))); 57 } 58 } 59 table.close(); 60 } 61 62 public static Configuration getConfig(){ 63 Configuration configuration = new Configuration(); 64 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 65 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 66 return configuration; 67 } 68 } 2016-12-10 17:58:16,226 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x7efa391e connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 17:58:16,265 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 17:58:16,267 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 17:58:16,267 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 17:58:16,267 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 17:58:16,268 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 17:58:16,268 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 17:58:16,270 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 17:58:16,271 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 17:58:16,271 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 17:58:16,271 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 17:58:16,272 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 17:58:16,272 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 17:58:16,272 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 17:58:16,273 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 17:58:16,273 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 17:58:16,278 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0x7efa391e0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 17:58:16,408 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopSlave1/192.168.80.11:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 17:58:16,438 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopSlave1/192.168.80.11:2181, initiating session 2016-12-10 17:58:16,474 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopSlave1/192.168.80.11:2181, sessionid = 0x2587338ecb40027, negotiated timeout = 40000 org.apache.hadoop.hbase.client.ClientScanner@634be93a keyvalues={row_01/f:col/1478102698687/Put/vlen=5/seqid=0, row_01/f:name/1478104345828/Put/vlen=4/seqid=0} family:f col:col valuemaizi keyvalues={row_01/f:col/1478102698687/Put/vlen=5/seqid=0, row_01/f:name/1478104345828/Put/vlen=4/seqid=0} family:f col:name valueAndy keyvalues={row_02/f:name/1478104477628/Put/vlen=5/seqid=0} family:f col:name valueAndy2 2016-12-10 17:58:17,994 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x2587338ecb40027 2016-12-10 17:58:18,001 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x2587338ecb40027 closed 2016-12-10 17:58:18,001 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 代码解释一 1 Scan scan = new Scan(); 2 scan.setStartRow(Bytes.toBytes("row_01"));//包含开始行键 3 scan.setStopRow(Bytes.toBytes("row_03"));//不包含结束行键 4 ResultScanner rst = table.getScanner(scan); 5 System.out.println(rst.toString()); 6 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 7 for(Cell cell:next.rawCells()){ 8 System.out.println(next.toString()); 9 System.out.println("family:" + Bytes.toString(CellUtil.cloneFamily(cell))); 10 System.out.println("col:" + Bytes.toString(CellUtil.cloneQualifier(cell))); 11 System.out.println("value" + Bytes.toString(CellUtil.cloneValue(cell))); 12 } 13 } 14 table.close(); 代码解释二 1 Scan scan = new Scan(); 2 scan.setStartRow(Bytes.toBytes("row_01"));//包含开始行键 3 scan.setStopRow(Bytes.toBytes("row_03"));//不包含结束行键 4 ResultScanner rst = table.getScanner(scan); 5 System.out.println(rst.toString()); 6 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 7 for(Cell cell:next.rawCells()){ 8 System.out.println(next.toString()); 9 System.out.println("family:" + Bytes.toString(CellUtil.cloneFamily(cell))); 10 System.out.println("col:" + Bytes.toString(CellUtil.cloneQualifier(cell))); 11 System.out.println("value" + Bytes.toString(CellUtil.cloneValue(cell))); 12 } 13 } 14 table.close(); 1 package zhouls.bigdata.HbaseProject.Test1; 2 3 import javax.xml.transform.Result; 4 5 import org.apache.hadoop.conf.Configuration; 6 import org.apache.hadoop.hbase.Cell; 7 import org.apache.hadoop.hbase.CellUtil; 8 import org.apache.hadoop.hbase.HBaseConfiguration; 9 import org.apache.hadoop.hbase.TableName; 10 import org.apache.hadoop.hbase.client.Delete; 11 import org.apache.hadoop.hbase.client.Get; 12 import org.apache.hadoop.hbase.client.HTable; 13 import org.apache.hadoop.hbase.client.Put; 14 import org.apache.hadoop.hbase.client.ResultScanner; 15 import org.apache.hadoop.hbase.client.Scan; 16 import org.apache.hadoop.hbase.util.Bytes; 17 18 public class HBaseTest { 19 public static void main(String[] args) throws Exception { 20 HTable table = new HTable(getConfig(),TableName.valueOf("test_table"));//表名是test_table 21 // Put put = new Put(Bytes.toBytes("row_04"));//行键是row_04 22 // put.add(Bytes.toBytes("f"),Bytes.toBytes("name"),Bytes.toBytes("Andy1"));//列簇是f，列修饰符是name，值是Andy0 23 // put.add(Bytes.toBytes("f2"),Bytes.toBytes("name"),Bytes.toBytes("Andy3"));//列簇是f2，列修饰符是name，值是Andy3 24 // table.put(put); 25 // table.close(); 26 27 // Get get = new Get(Bytes.toBytes("row_04")); 28 // get.addColumn(Bytes.toBytes("f1"), Bytes.toBytes("age"));如现在这样，不指定，默认把所有的全拿出来 29 // org.apache.hadoop.hbase.client.Result rest = table.get(get); 30 // System.out.println(rest.toString()); 31 // table.close(); 32 33 // Delete delete = new Delete(Bytes.toBytes("row_2")); 34 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("email")); 35 // delete.deleteColumn(Bytes.toBytes("f1"), Bytes.toBytes("name")); 36 // table.delete(delete); 37 // table.close(); 38 39 // Delete delete = new Delete(Bytes.toBytes("row_04")); 40 //// delete.deleteColumn(Bytes.toBytes("f"), Bytes.toBytes("name"));//deleteColumn是删除某一个列簇里的最新时间戳版本。 41 // delete.deleteColumns(Bytes.toBytes("f"), Bytes.toBytes("name"));//delete.deleteColumns是删除某个列簇里的所有时间戳版本。 42 // table.delete(delete); 43 // table.close(); 44 45 46 Scan scan = new Scan(); 47 scan.setStartRow(Bytes.toBytes("row_01"));//包含开始行键 48 scan.setStopRow(Bytes.toBytes("row_03"));//不包含结束行键 49 scan.addColumn(Bytes.toBytes("f"), Bytes.toBytes("name")); 50 ResultScanner rst = table.getScanner(scan);//整个循环 51 System.out.println(rst.toString()); 52 for (org.apache.hadoop.hbase.client.Result next = rst.next();next !=null;next = rst.next() ){ 53 for(Cell cell:next.rawCells()){//某个row key下的循坏 54 System.out.println(next.toString()); 55 System.out.println("family:" + Bytes.toString(CellUtil.cloneFamily(cell))); 56 System.out.println("col:" + Bytes.toString(CellUtil.cloneQualifier(cell))); 57 System.out.println("value" + Bytes.toString(CellUtil.cloneValue(cell))); 58 } 59 } 60 table.close(); 61 } 62 63 public static Configuration getConfig(){ 64 Configuration configuration = new Configuration(); 65 // conf.set("hbase.rootdir","hdfs：HadoopMaster:9000/hbase"); 66 configuration.set("hbase.zookeeper.quorum", "HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181"); 67 return configuration; 68 } 69 } 2016-12-10 18:09:00,581 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0xbe4535e connecting to ZooKeeper ensemble=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 2016-12-10 18:09:00,600 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-12-10 18:09:00,601 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=WIN-BQOBV63OBNM 2016-12-10 18:09:00,601 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_51 2016-12-10 18:09:00,601 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation 2016-12-10 18:09:00,601 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=C:\Program Files\Java\jdk1.7.0_51\jre 2016-12-10 18:09:00,601 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=D:\Code\MyEclipseJavaCode\HbaseProject\bin;D:\SoftWare\hbase-1.2.3\lib\activation-1.1.jar;D:\SoftWare\hbase-1.2.3\lib\aopalliance-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-i18n-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\SoftWare\hbase-1.2.3\lib\api-asn1-api-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\api-util-1.0.0-M20.jar;D:\SoftWare\hbase-1.2.3\lib\asm-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\avro-1.7.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-1.7.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-beanutils-core-1.8.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-cli-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-codec-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\commons-collections-3.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-compress-1.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-configuration-1.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-daemon-1.0.13.jar;D:\SoftWare\hbase-1.2.3\lib\commons-digester-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\commons-el-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\commons-httpclient-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-io-2.4.jar;D:\SoftWare\hbase-1.2.3\lib\commons-lang-2.6.jar;D:\SoftWare\hbase-1.2.3\lib\commons-logging-1.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math-2.2.jar;D:\SoftWare\hbase-1.2.3\lib\commons-math3-3.1.1.jar;D:\SoftWare\hbase-1.2.3\lib\commons-net-3.1.jar;D:\SoftWare\hbase-1.2.3\lib\disruptor-3.3.0.jar;D:\SoftWare\hbase-1.2.3\lib\findbugs-annotations-1.3.9-1.jar;D:\SoftWare\hbase-1.2.3\lib\guava-12.0.1.jar;D:\SoftWare\hbase-1.2.3\lib\guice-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\guice-servlet-3.0.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-annotations-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-auth-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-hdfs-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-app-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-jobclient-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-mapreduce-client-shuffle-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-api-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-client-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hadoop-yarn-server-common-2.5.1.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-annotations-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-client-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-common-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-examples-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-external-blockcache-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop2-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-hadoop-compat-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-it-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-prefix-tree-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-procedure-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-protocol-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-resource-bundle-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-rest-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-server-1.2.3-tests.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-shell-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\hbase-thrift-1.2.3.jar;D:\SoftWare\hbase-1.2.3\lib\htrace-core-3.1.0-incubating.jar;D:\SoftWare\hbase-1.2.3\lib\httpclient-4.2.5.jar;D:\SoftWare\hbase-1.2.3\lib\httpcore-4.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-core-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-jaxrs-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-mapper-asl-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jackson-xc-1.9.13.jar;D:\SoftWare\hbase-1.2.3\lib\jamon-runtime-2.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-compiler-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\jasper-runtime-5.5.23.jar;D:\SoftWare\hbase-1.2.3\lib\javax.inject-1.jar;D:\SoftWare\hbase-1.2.3\lib\java-xmlbuilder-0.4.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-api-2.2.2.jar;D:\SoftWare\hbase-1.2.3\lib\jaxb-impl-2.2.3-1.jar;D:\SoftWare\hbase-1.2.3\lib\jcodings-1.0.8.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-client-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-core-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-guice-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-json-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jersey-server-1.9.jar;D:\SoftWare\hbase-1.2.3\lib\jets3t-0.9.0.jar;D:\SoftWare\hbase-1.2.3\lib\jettison-1.3.3.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-sslengine-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\jetty-util-6.1.26.jar;D:\SoftWare\hbase-1.2.3\lib\joni-2.1.2.jar;D:\SoftWare\hbase-1.2.3\lib\jruby-complete-1.6.8.jar;D:\SoftWare\hbase-1.2.3\lib\jsch-0.1.42.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\jsp-api-2.1-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\junit-4.12.jar;D:\SoftWare\hbase-1.2.3\lib\leveldbjni-all-1.8.jar;D:\SoftWare\hbase-1.2.3\lib\libthrift-0.9.3.jar;D:\SoftWare\hbase-1.2.3\lib\log4j-1.2.17.jar;D:\SoftWare\hbase-1.2.3\lib\metrics-core-2.2.0.jar;D:\SoftWare\hbase-1.2.3\lib\netty-all-4.0.23.Final.jar;D:\SoftWare\hbase-1.2.3\lib\paranamer-2.3.jar;D:\SoftWare\hbase-1.2.3\lib\protobuf-java-2.5.0.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5.jar;D:\SoftWare\hbase-1.2.3\lib\servlet-api-2.5-6.1.14.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-api-1.7.7.jar;D:\SoftWare\hbase-1.2.3\lib\slf4j-log4j12-1.7.5.jar;D:\SoftWare\hbase-1.2.3\lib\snappy-java-1.0.4.1.jar;D:\SoftWare\hbase-1.2.3\lib\spymemcached-2.11.6.jar;D:\SoftWare\hbase-1.2.3\lib\xmlenc-0.52.jar;D:\SoftWare\hbase-1.2.3\lib\xz-1.0.jar;D:\SoftWare\hbase-1.2.3\lib\zookeeper-3.4.6.jar 2016-12-10 18:09:00,603 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\SoftWare\MATLAB R2013a\runtime\win64;D:\SoftWare\MATLAB R2013a\bin;C:\Program Files (x86)\IDM Computer Solutions\UltraCompare;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Program Files\Java\jdk1.7.0_51\jre\bin;D:\SoftWare\apache-ant-1.9.0\bin;HADOOP_HOME\bin;D:\SoftWare\apache-maven-3.3.9\bin;D:\SoftWare\Scala\bin;D:\SoftWare\Scala\jre\bin;%MYSQL_HOME\bin;D:\SoftWare\MySQL Server\MySQL Server 5.0\bin;D:\SoftWare\apache-tomcat-7.0.69\bin;%C:\Windows\System32;%C:\Windows\SysWOW64;D:\SoftWare\SSH Secure Shell;. 2016-12-10 18:09:00,603 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\ 2016-12-10 18:09:00,604 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA> 2016-12-10 18:09:00,604 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Windows 7 2016-12-10 18:09:00,604 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64 2016-12-10 18:09:00,604 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=6.1 2016-12-10 18:09:00,605 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=Administrator 2016-12-10 18:09:00,605 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=C:\Users\Administrator 2016-12-10 18:09:00,605 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=D:\Code\MyEclipseJavaCode\HbaseProject 2016-12-10 18:09:00,609 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181 sessionTimeout=180000 watcher=hconnection-0xbe4535e0x0, quorum=HadoopMaster:2181,HadoopSlave1:2181,HadoopSlave2:2181, baseZNode=/hbase 2016-12-10 18:09:00,717 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server HadoopMaster/192.168.80.10:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-10 18:09:00,729 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to HadoopMaster/192.168.80.10:2181, initiating session 2016-12-10 18:09:01,013 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server HadoopMaster/192.168.80.10:2181, sessionid = 0x15825dedcf40031, negotiated timeout = 40000 org.apache.hadoop.hbase.client.ClientScanner@12f8288e keyvalues={row_01/f:name/1478104345828/Put/vlen=4/seqid=0} family:f col:name valueAndy keyvalues={row_02/f:name/1478104477628/Put/vlen=5/seqid=0} family:f col:name valueAndy2 2016-12-10 18:09:02,563 INFO [org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x15825dedcf40031 2016-12-10 18:09:03,395 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15825dedcf40031 closed 2016-12-10 18:09:03,419 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down 本文转自大数据躺过的坑博客园博客，原文链接：http://www.cnblogs.com/zlslch/p/6156343.html，如需转载请自行联系原作者

2017-11-11

Hadoop入门进阶课程7--Pig介绍、安装与应用案例

2017-11-09

Hadoop入门进阶课程9--Mahout介绍、安装与应用案例

本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接，博主为石山园，博客地址为http://www.cnblogs.com/shishanyuan。该系列课程是应邀实验楼整理编写的，这里需要赞一下实验楼提供了学习的新方式，可以边看博客边上机实验，课程地址为https://www.shiyanlou.com/courses/237 【注】该系列所使用到安装包、测试数据和代码均可在百度网盘下载，具体地址为http://pan.baidu.com/s/10PnDs，下载该PDF文件 1、搭建环境部署节点操作系统为CentOS，防火墙和SElinux禁用，创建了一个shiyanlou用户并在系统根目录下创建/app目录，用于存放Hadoop等组件运行包。因为该目录用于安装hadoop等组件程序，用户对shiyanlou必须赋予rwx权限（一般做法是root用户在根目录下创建/app目录，并修改该目录拥有者为shiyanlou(chown–R shiyanlou:shiyanlou /app）。 Hadoop搭建环境： l虚拟机操作系统：CentOS6.664位，单核，1G内存 lJDK：1.7.0_55 64位 lHadoop：1.1.2 2、Mahout介绍 Mahout是Apache Software Foundation（ASF）旗下的一个开源项目，提供一些可扩展的机器学习领域经典算法的实现，旨在帮助开发人员更加方便快捷地创建智能应用程序。AMahout包含许多实现，包括聚类、分类、推荐过滤、频繁子项挖掘。此外，通过使用Apache Hadoop库，Mahout可以有效地扩展到云中。 Mahout的意思是大象的饲养者及驱赶者。Mahout这个名称来源于该项目（有时）使用Apache Hadoop—其徽标上有一头黄色的大象 —来实现可伸缩性和容错性。 Mahout项目是由Apache Lucene（开源搜索）社区中对机器学习感兴趣的一些成员发起的，他们希望建立一个可靠、文档翔实、可伸缩的项目，在其中实现一些常见的用于集群和分类的机器学习算法。该社区最初基于Ng et al.的文章 “Map-Reduce for Machine Learning on Multicore”（见参考资料），但此后在发展中又并入了更多广泛的机器学习方法。Mahout的目标还包括： l建立一个用户和贡献者社区，使代码不必依赖于特定贡献者的参与或任何特定公司和大学的资金。 l专注于实际用例，这与高新技术研究及未经验证的技巧相反。 l提供高质量文章和示例 3、搭建Mahout环境 3.1部署过程 3.1.1下载Mahout 在Apache下载最新的Mahout软件包，点击下载会推荐最快的镜像站点，以下为下载地址：http://archive.apache.org/dist/mahout/0.6/ 也可以在/home/shiyanlou/install-pack目录中找到该安装包，解压该安装包并把该安装包复制到/app目录中 cd /home/shiyanlou/install-pack tar -xzf mahout-distribution-0.6.tar.gz mv mahout-distribution-0.6 /app/mahout-0.6 3.1.2设置环境变量使用如下命令编辑/etc/profile文件： sudo vi /etc/profile 声明mahout的home路径和在path加入bin的路径： export MAHOUT_HOME=/app/mahout-0.6 export MAHOUT_CONF_DIR=/app/mahout-0.6/conf export PATH=$PATH:$MAHOUT_HOME/bin 编译配置文件/etc/profile，并确认生效 source /etc/profile echo $PATH 3.1.3验证安装完成重新登录终端，确保hadoop集群启动，键入mahout --help命令，检查Mahout是否安装完好，看是否列出了一些算法： mahout --help 3.2测试例子 3.2.1下载测试数据下载一个文件synthetic_control.data，下载地址http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data，也可以在/home/shiyanlou/install-pack/class9目录中找到该测试数据文件，把这个文件放在$MAHOUT_HOME/testdata目录下 cd /home/shiyanlou/install-pack/class9 mkdir /app/mahout-0.6/testdata mv synthetic_control.data /app/mahout-0.6/testdata 3.2.2启动Hadoop 通过下面命令启动hadoop并通过jps查看进程 cd /app/hadoop-1.1.2/bin ./start-all.sh jps 3.2.3使用kmeans算法使用如下命令进行kmeans算法测试： cd /app/mahout-0.6/ mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job 这里需要说明下，当你看到下面的代码时以为是错的，其实不是，原因：MAHOUT_LOCAL：设置是否本地运行，如果设置该参数就不会在hadoop运行了，一旦设置这个参数那HADOOP_CONF_DIR和HADOOP_HOME两个参数就自动失效了。 MAHOUT_LOCAL is not set, so we don't add HADOOP_CONF_DIR to classpath. no HADOOP_HOME set , running locally 3.2.4查看结果结果会在根目录建立output新文件夹，如果下图结果表示mahout安装正确且运行正常： cd /app/mahout-0.6/output ll 4、测试例子：运行20newsgroup 4.1算法流程朴素贝叶斯分类是一种十分简单的分类算法，朴素贝叶斯的思想基础是这样的：对于给出的待分类项，求解在此项出现的条件下各个类别出现的概率哪个最大，就认为此待分类项属于哪个类别。这二十个新闻组数据集合是收集大约20,000新闻组文档，均匀的分布在20个不同的集合。这20个新闻组集合采集最近流行的数据集合到文本程序中作为实验，根据机器学习技术。例如文本分类，文本聚集。我们将使用Mahout的Bayes Classifier创造一个模型，它将一个新文档分类到这20个新闻组集合范例演示 4.2实现过程（mahout 0.6版本） 4.2.1下载数据并解压数据下载20Newsgroups数据集，地址为http://qwone.com/~jason/20Newsgroups/，下载20news-bydate.tar.gz数据包，也可以在/home/shiyanlou/install-pack/class9目录中找到该测试数据文件：解压20news-bydate.tar.gz数据包，解压后可以看到两个文件夹，分别为训练原始数据和测试原始数据： cd /home/shiyanlou/install-pack/class9 tar -xzf 20news-bydate.tar.gz 在mahout根目录下建data文件夹，然后把20news训练原始数据和测试原始数据迁移到该文件夹下： mkdir /app/mahout-0.6/data mv 20news-bydate-t* /app/mahout-0.6/data ll /app/mahout-0.6/data 4.2.2建立训练集通过如下命令建立训练集，训练的数据在20news-bydate-train目录中，输出的训练集目录为bayes-train-input： cd /app/mahout-0.6 mahout org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups \ -p /app/mahout-0.6/data/20news-bydate-train \ -o /app/mahout-0.6/data/bayes-train-input \ -a org.apache.mahout.vectorizer.DefaultAnalyzer \ -c UTF-8 4.2.3建立测试集通过如下命令建立训练集，训练的数据在20news-bydate-test目录中，输出的训练集目录为bayes-test-input： mahout org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups \ -p /app/mahout-0.6/data/20news-bydate-test \ -o /app/mahout-0.6/data/bayes-test-input \ -a org.apache.mahout.vectorizer.DefaultAnalyzer \ -c UTF-8 4.2.4上传数据到HDFS 在HDFS中新建/class9/20news文件夹，把生成的训练集和测试集上传到HDFS的/class9/20news目录中： hadoop fs -mkdir /class9/20news hadoop fs -put /app/mahout-0.6/data/bayes-train-input /class9/20news hadoop fs -put /app/mahout-0.6/data/bayes-test-input /class9/20news hadoop fs -ls /class9/20news hadoop fs -ls /class9/20news/bayes-test-input 4.2.5训练贝叶斯分类器使用trainclassifier类训练在HDFS中/class9/20news/bayes-train-input的数据，生成的模型放到/class9/ 20news/newsmodel目录中： mahout trainclassifier \ -i /class9/20news/bayes-train-input \ -o /class9/20news/newsmodel \ -type cbayes \ -ng 2 \ -source hdfs 4.2.6观察训练作业运行过程注：实验楼为命令行界面，无法观测到该步骤界面，以下描述仅做参考在训练过程中在JobTracker页面观察运行情况，链接地址为http://**.***.**.***:50030/jobtracker.jsp，训练任务四个作业，大概运行了15分钟左右：点击查看具体作业信息 map运行情况作业运行情况 4.2.7查看生成模型通过如下命令查看模型内容： hadoop fs -ls /class9/20news hadoop fs -ls /class9/20news/newsmodel hadoop fs -ls /class9/20news/newsmodel/trainer-tfIdf 4.2.8测试贝叶斯分类器使用testclassifier类训练在HDFS中./20news/bayestest-input的数据，使用的模型路径为./ 20news/newsmodel： mahout testclassifier \ -m /class9/20news/newsmodel \ -d /class9/20news/bayes-test-input \ -type cbayes \ -ng 2 \ -source hdfs \ -method mapreduce 4.2.9观察训练作业运行过程注：实验楼为命令行界面，无法观测到该步骤界面，以下描述仅做参考在执行过程中在JobTracker页面观察运行情况，链接地址为http://hadoop:50030/jobtracker.jsp，训练任务1个作业，大概运行了5分钟左右：作业的基本信息 map运行情况 reduce运行情况 4.2.10查看结果这个混合矩阵的意思说明：上述a到u分别是代表了有20类别，这就是我们之前给的20个输入文件，列中的数据说明每个类别中被分配到的字节个数，classified说明应该被分配到的总数 38100009100010020100300|398a = rec.motorcycles 意思为rec.motorcycles本来是属于a，有381篇文档被划为了a类，这个是正确的数据，其它的分别表示划到b~u类中的数目。我们可以看到其正确率为381/398=0.9573 ,可见其正确率还是很高的了。 4.3实现过程（mahout 0.7+版本）在0.7版本的安装目录下$MAHOUT_HOME/examples/bin下有个脚本文件classifu-20newsgroups.sh，这个脚本中执行过程是和前面分布执行结果是一致的，只不过将各个API用shell脚本封装到一起了。从0.7版本开始，Mahout移除了命令行调用的API：prepare20newsgroups、trainclassifier和testclassifier，只能通过shell脚本执行。执行$MAHOUT_HOME/examples/bin/classify-20newsgroups.sh四个选项中选择第一个选项，执行结果如下： 5、问题解决 5.1使用mahout0.7+版本对20Newsgroup数据建立训练集时出错使用如下命令对20Newsgroupt数据建立训练集时： mahout org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups \ -p /app/mahout-0.9/data/20news-bydate-train \ -o /app/mahout-0.9/data/bayes-train-input \ -a org.apache.mahout.vectorizer.DefaultAnalyzer\ -c UTF-8 出现如下错误，原因在于从0.7版本开始，Mahout移除了命令行调用的prepare20newsgroups、trainclassifier和testclassifier API，只能通过shell脚本执行$MAHOUT_HOME/examples/bin/classify-20newsgroups.sh进行 14/12/7 21:31:35 WARN driver.MahoutDriver: Unable to add class: org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups 14/12/7 21:31:35 WARN driver.MahoutDriver: No org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups.props found on classpath, will use command-line arguments only Unknown program 'org.apache.mahout.classifier.bayes.PrepareTwentyNewsgroups' chosen. Valid program names are: arff.vector: : Generate Vectors from an ARFF file or directory baumwelch: : Baum-Welch algorithm for unsupervised HMM training ....... 本文转自shishanyuan博客园博客，原文链接： http://www.cnblogs.com/shishanyuan/p/4648258.html，如需转载请自行联系原作者

2017-11-08

资源下载

更多资源

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。

WebStorm

WebStorm 是jetbrains公司旗下一款JavaScript 开发工具。目前已经被广大中国JS开发者誉为“Web前端开发神器”、“最强大的HTML5编辑器”、“最智能的JavaScript IDE”等。与IntelliJ IDEA同源，继承了IntelliJ IDEA强大的JS部分的功能。

精选列表

LeadTools Android 入门教学——运行第一个Android Demo

Hadoop MapReduce编程 API入门系列之倒排索引（二十四）

Hadoop MapReduce编程 API入门系列之MapReduce多种输入格式（十七）

Hadoop MapReduce编程 API入门系列之二次排序（十六）

Hadoop入门进阶课程12--Flume介绍、安装与应用案例

Provisioning Services 7.6 入门到精通系列之二：基础架构环境

Android GIS开发系列-- 入门季（4） GraphicsLayer的点击查询要素

Provisioning Services 7.6 入门到精通系列之一：PVS前期规划

从入门到精通AD中的5种操作主机

Android GIS开发系列-- 入门季（3） GraphicsLayer添加点、线、面

Hadoop MapReduce编程 API入门系列之网页排序（二十八）

HBase编程 API入门系列之delete（客户端而言）（3）

HBase编程 API入门系列之get（客户端而言）（2）

HBase编程 API入门系列之put（客户端而言）（1）

OpenStack入门修炼之如何创建生产使用的openstack镜像（16）

OpenStack入门修炼之KVM虚拟机介绍和管理（4）

OpenStack入门修炼之KVM性能优化及IO缓存介绍（5）

HBase编程 API入门系列之scan（客户端而言）（5）

Hadoop入门进阶课程7--Pig介绍、安装与应用案例

Hadoop入门进阶课程9--Mahout介绍、安装与应用案例

资源下载

Mario

Rocky Linux

Sublime Text

WebStorm

欢迎您来访！