Hadoop MapReduce之wordcount(词频统计)-低调大师

Hadoop MapReduce之wordcount(词频统计)

2016-02-27 687

1.创建test.log

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 mapreduce]# more /tmp/test.log
1
2
3
a
b
a
v
a a a
abc
我是谁
%……
%

2.hadoop创建目录及上传

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 ~]# hadoop fs -mkdir /testdir
16/02/28 19:40:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@sht-sgmhadoopnn-01 ~]# hadoop fs -put /tmp/test.log /testdir/
16/02/28 19:40:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

3.查看官方封装好的函数,我们选取wordcount

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 ~]#cd /hadoop/hadoop-2.7.2/share/hadoop/mapreduce
[root@sht-sgmhadoopnn-01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.2.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

4.运行wordcount
# hadoop jar hadoop-mapreduce-examples-2.7.2.jar wordcount /testdir /out1
# 官方模板jar包函数输入目录输出目录(未创建)

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.2.jar wordcount /testdir /out1
16/02/28 19:40:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/28 19:40:53 INFO input.FileInputFormat: Total input paths to process : 1
16/02/28 19:40:53 INFO mapreduce.JobSubmitter: number of splits:1
16/02/28 19:40:53 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1456590271264_0002
16/02/28 19:40:54 INFO impl.YarnClientImpl: Submitted application application_1456590271264_0002
16/02/28 19:40:54 INFO mapreduce.Job: The url to track the job: http://sht-sgmhadoopnn-01:8088/proxy/application_1456590271264_0002/
16/02/28 19:40:54 INFO mapreduce.Job: Running job: job_1456590271264_0002
16/02/28 19:41:04 INFO mapreduce.Job: Job job_1456590271264_0002 running in uber mode : false
16/02/28 19:41:04 INFO mapreduce.Job: map 0% reduce 0%
16/02/28 19:41:12 INFO mapreduce.Job: map 100% reduce 0%
16/02/28 19:41:21 INFO mapreduce.Job: map 100% reduce 100%
16/02/28 19:41:22 INFO mapreduce.Job: Job job_1456590271264_0002 completed successfully
16/02/28 19:41:22 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=102
FILE: Number of bytes written=244621
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=142
HDFS: Number of bytes written=56
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5537
Total time spent by all reduces in occupied slots (ms)=6555
Total time spent by all map tasks (ms)=5537
Total time spent by all reduce tasks (ms)=6555
Total vcore-milliseconds taken by all map tasks=5537
Total vcore-milliseconds taken by all reduce tasks=6555
Total megabyte-milliseconds taken by all map tasks=5669888
Total megabyte-milliseconds taken by all reduce tasks=6712320
Map-Reduce Framework
Map input records=12
Map output records=14
Map output bytes=100
Map output materialized bytes=102
Input split bytes=98
Combine input records=14
Combine output records=10
Reduce input groups=10
Reduce shuffle bytes=102
Reduce input records=10
Reduce output records=10
Spilled Records=20
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=79
CPU time spent (ms)=2560
Physical memory (bytes) snapshot=445992960
Virtual memory (bytes) snapshot=1775263744
Total committed heap usage (bytes)=306184192
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=44
File Output Format Counters
Bytes Written=56
You have mail in /var/spool/mail/root
[root@sht-sgmhadoopnn-01 mapreduce]#

5.验证wordcount，词频统计

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 mapreduce]# hadoop fs -ls /out1
16/02/28 19:43:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 3 root supergroup 0 2016-02-28 19:41 /out1/_SUCCESS
-rw-r--r-- 3 root supergroup 56 2016-02-28 19:41 /out1/part-r-00000
[root@sht-sgmhadoopnn-01 mapreduce]# hadoop fs -text /out1/part-r-00000
16/02/28 19:43:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
% 1
%…… 1
1 1
2 1
3 1
a 5
abc 1
b 1
v 1
我是谁 1
You have mail in /var/spool/mail/root

微信关注我们

原文链接：https://yq.aliyun.com/articles/578893

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

Hive事物和锁管理

摘自Hive技术文档，锁管理：https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-LockManager 有几个参数需要修改下，使得Hive支持事物和并发； Lock Manager A new lock manager has also been added to Hive, the DbLockManager. This lock manager stores all lock information in the metastore. In addition all transactions are stored in the metastore. This means that transactions and locks are durable in the face of server failure. To avoid clients dying and leaving transaction or locks dangling, a heartbeat i...

2016-02-28

1153

在《Spark源码分析之七：Task运行（一）》一文中，我们详细叙述了Task运行的整体流程，最终Task被传输到Executor上，启动一个对应的TaskRunner线程，并且在线程池中被调度执行。继而，我们对TaskRunner的run()方法进行了详细的分析，总结出了其内Task执行的三个主要步骤： Step1：Task及其运行时需要的辅助对象构造，主要包括： 1、当前线程设置上下文类加载器； 2、获取序列化器ser； 3、更新任务状态TaskState； 4、计算垃圾回收时间； 5、反序列化得到Task运行的jar、文件、Task对象二进制数据； 6、反序列化Task对象二进制数据得到Task对象； 7、设置任务内存管理器； Step2：Task运行：调用Task的run()方法，真正执行Task，并获得运行结果value Step3：Task运行结果处理： 1、序列化Task运行...

2016-02-28

592

资源下载

更多资源

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。