Hadoop MapReduce之wordcount(词频统计)-低调大师

Hadoop MapReduce之wordcount(词频统计)

2016-02-27 705

1.创建test.log

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 mapreduce]# more /tmp/test.log
1
2
3
a
b
a
v
a a a
abc
我是谁
%……
%

2.hadoop创建目录及上传

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 ~]# hadoop fs -mkdir /testdir
16/02/28 19:40:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@sht-sgmhadoopnn-01 ~]# hadoop fs -put /tmp/test.log /testdir/
16/02/28 19:40:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

3.查看官方封装好的函数,我们选取wordcount

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 ~]#cd /hadoop/hadoop-2.7.2/share/hadoop/mapreduce
[root@sht-sgmhadoopnn-01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.2.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

4.运行wordcount
# hadoop jar hadoop-mapreduce-examples-2.7.2.jar wordcount /testdir /out1
# 官方模板jar包函数输入目录输出目录(未创建)

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.2.jar wordcount /testdir /out1
16/02/28 19:40:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/28 19:40:53 INFO input.FileInputFormat: Total input paths to process : 1
16/02/28 19:40:53 INFO mapreduce.JobSubmitter: number of splits:1
16/02/28 19:40:53 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1456590271264_0002
16/02/28 19:40:54 INFO impl.YarnClientImpl: Submitted application application_1456590271264_0002
16/02/28 19:40:54 INFO mapreduce.Job: The url to track the job: http://sht-sgmhadoopnn-01:8088/proxy/application_1456590271264_0002/
16/02/28 19:40:54 INFO mapreduce.Job: Running job: job_1456590271264_0002
16/02/28 19:41:04 INFO mapreduce.Job: Job job_1456590271264_0002 running in uber mode : false
16/02/28 19:41:04 INFO mapreduce.Job: map 0% reduce 0%
16/02/28 19:41:12 INFO mapreduce.Job: map 100% reduce 0%
16/02/28 19:41:21 INFO mapreduce.Job: map 100% reduce 100%
16/02/28 19:41:22 INFO mapreduce.Job: Job job_1456590271264_0002 completed successfully
16/02/28 19:41:22 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=102
FILE: Number of bytes written=244621
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=142
HDFS: Number of bytes written=56
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5537
Total time spent by all reduces in occupied slots (ms)=6555
Total time spent by all map tasks (ms)=5537
Total time spent by all reduce tasks (ms)=6555
Total vcore-milliseconds taken by all map tasks=5537
Total vcore-milliseconds taken by all reduce tasks=6555
Total megabyte-milliseconds taken by all map tasks=5669888
Total megabyte-milliseconds taken by all reduce tasks=6712320
Map-Reduce Framework
Map input records=12
Map output records=14
Map output bytes=100
Map output materialized bytes=102
Input split bytes=98
Combine input records=14
Combine output records=10
Reduce input groups=10
Reduce shuffle bytes=102
Reduce input records=10
Reduce output records=10
Spilled Records=20
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=79
CPU time spent (ms)=2560
Physical memory (bytes) snapshot=445992960
Virtual memory (bytes) snapshot=1775263744
Total committed heap usage (bytes)=306184192
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=44
File Output Format Counters
Bytes Written=56
You have mail in /var/spool/mail/root
[root@sht-sgmhadoopnn-01 mapreduce]#

5.验证wordcount，词频统计

点击(此处)折叠或打开

[root@sht-sgmhadoopnn-01 mapreduce]# hadoop fs -ls /out1
16/02/28 19:43:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 3 root supergroup 0 2016-02-28 19:41 /out1/_SUCCESS
-rw-r--r-- 3 root supergroup 56 2016-02-28 19:41 /out1/part-r-00000
[root@sht-sgmhadoopnn-01 mapreduce]# hadoop fs -text /out1/part-r-00000
16/02/28 19:43:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
% 1
%…… 1
1 1
2 1
3 1
a 5
abc 1
b 1
v 1
我是谁 1
You have mail in /var/spool/mail/root

微信关注我们

原文链接：https://yq.aliyun.com/articles/578893

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

Hive事物和锁管理

摘自Hive技术文档，锁管理：https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-LockManager 有几个参数需要修改下，使得Hive支持事物和并发； Lock Manager A new lock manager has also been added to Hive, the DbLockManager. This lock manager stores all lock information in the metastore. In addition all transactions are stored in the metastore. This means that transactions and locks are durable in the face of server failure. To avoid clients dying and leaving transaction or locks dangling, a heartbeat i...

2016-02-28

1182

在《Spark源码分析之七：Task运行（一）》一文中，我们详细叙述了Task运行的整体流程，最终Task被传输到Executor上，启动一个对应的TaskRunner线程，并且在线程池中被调度执行。继而，我们对TaskRunner的run()方法进行了详细的分析，总结出了其内Task执行的三个主要步骤： Step1：Task及其运行时需要的辅助对象构造，主要包括： 1、当前线程设置上下文类加载器； 2、获取序列化器ser； 3、更新任务状态TaskState； 4、计算垃圾回收时间； 5、反序列化得到Task运行的jar、文件、Task对象二进制数据； 6、反序列化Task对象二进制数据得到Task对象； 7、设置任务内存管理器； Step2：Task运行：调用Task的run()方法，真正执行Task，并获得运行结果value Step3：Task运行结果处理： 1、序列化Task运行...

2016-02-28

619

资源下载

更多资源

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。