热门搜索:
您现在的位置是：首页 > 文章详情

hadoop 文件操作和代码运行测试

日期：2017-11-01点击：517
1. 查看HDFS文件或目录
[xiaoyu@hadoop02 ~]$ cd hadoop-1.1.2
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -ls
[xiaoyu@hadoop02 hadoop-1.1.2]$ echo $?
0
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -lsr
[xiaoyu@hadoop02 hadoop-1.1.2]$ echo $?
0
[xiaoyu@hadoop02 hadoop-1.1.2]$
2. 新建HDFS目录或文件
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -mkdir TEST
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -lsr
drwxr-xr-x - xiaoyu supergroup 0 2013-09-10 14:08 /user/xiaoyu/TEST
3. 上传文件到HDFS目录中
本地建立一个文件，内容如下：
[xiaoyu@hadoop02 hadoop-1.1.2]$ touch test.txt
[xiaoyu@hadoop02 hadoop-1.1.2]$ vim test.txt
[xiaoyu@hadoop02 hadoop-1.1.2]$ cat test.txt
Hello, Hadoop !
你好, Hadoop ！
上传到HDFS的TEST目录中，可以使用-copyFromLocal参数，也可以使用-moveFromLocal，分别是从本地拷贝到和从本地剪切到HDFS目录。
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -moveFromLocal test.txt TEST
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -ls
Found 1 items
drwxr-xr-x - xiaoyu supergroup 0 2013-09-10 14:14 /user/xiaoyu/TEST
[xiaoyu@hadoop02 hadoop-1.1.2]$
4. 查看HDFS文件内容
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -cat TEST/test.txt
Hello, Hadoop !
你好, Hadoop ！
[xiaoyu@hadoop02 hadoop-1.1.2]$
5. 运行jar范例，统计文本单词词频
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop jar hadoop-examples-1.1.2.jar
wordcount TEST out
13/09/10 14:20:50 INFO input.FileInputFormat: Total input paths to process : 1
13/09/10 14:20:50 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/09/10 14:20:50 WARN snappy.LoadSnappy: Snappy native library not loaded
13/09/10 14:20:51 INFO mapred.JobClient: Running job: job_201309082325_0001
13/09/10 14:20:52 INFO mapred.JobClient: map 0% reduce 0%
13/09/10 14:21:03 INFO mapred.JobClient: map 100% reduce 0%
13/09/10 14:21:11 INFO mapred.JobClient: map 100% reduce 33%
13/09/10 14:21:13 INFO mapred.JobClient: map 100% reduce 100%
13/09/10 14:21:14 INFO mapred.JobClient: Job complete: job_201309082325_0001
13/09/10 14:21:14 INFO mapred.JobClient: Counters: 29
13/09/10 14:21:14 INFO mapred.JobClient: Job Counters
13/09/10 14:21:14 INFO mapred.JobClient: Launched reduce tasks=1
13/09/10 14:21:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=10619
13/09/10 14:21:14 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/09/10 14:21:14 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/09/10 14:21:14 INFO mapred.JobClient: Launched map tasks=1
13/09/10 14:21:14 INFO mapred.JobClient: Data-local map tasks=1
13/09/10 14:21:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=9864
13/09/10 14:21:14 INFO mapred.JobClient: File Output Format Counters
13/09/10 14:21:14 INFO mapred.JobClient: Bytes Written=38
13/09/10 14:21:14 INFO mapred.JobClient: FileSystemCounters
13/09/10 14:21:14 INFO mapred.JobClient: FILE_BYTES_READ=64
13/09/10 14:21:14 INFO mapred.JobClient: HDFS_BYTES_READ=146
13/09/10 14:21:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=109756
13/09/10 14:21:14 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=38
13/09/10 14:21:14 INFO mapred.JobClient: File Input Format Counters
13/09/10 14:21:14 INFO mapred.JobClient: Bytes Read=35
13/09/10 14:21:14 INFO mapred.JobClient: Map-Reduce Framework
13/09/10 14:21:14 INFO mapred.JobClient: Map output materialized bytes=64
13/09/10 14:21:14 INFO mapred.JobClient: Map input records=2
13/09/10 14:21:14 INFO mapred.JobClient: Reduce shuffle bytes=64
13/09/10 14:21:14 INFO mapred.JobClient: Spilled Records=10
13/09/10 14:21:14 INFO mapred.JobClient: Map output bytes=59
13/09/10 14:21:14 INFO mapred.JobClient: Total committed heap usage (bytes)=189464576
13/09/10 14:21:14 INFO mapred.JobClient: CPU time spent (ms)=4420
13/09/10 14:21:14 INFO mapred.JobClient: Combine input records=6
13/09/10 14:21:14 INFO mapred.JobClient: SPLIT_RAW_BYTES=111
13/09/10 14:21:14 INFO mapred.JobClient: Reduce input records=5
13/09/10 14:21:14 INFO mapred.JobClient: Reduce input groups=5
13/09/10 14:21:14 INFO mapred.JobClient: Combine output records=5
13/09/10 14:21:14 INFO mapred.JobClient: Physical memory (bytes) snapshot=281489408
13/09/10 14:21:14 INFO mapred.JobClient: Reduce output records=5
13/09/10 14:21:14 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1542262784
13/09/10 14:21:14 INFO mapred.JobClient: Map output records=6
6. 检查步骤5运行结果
[xiaoyu@hadoop02 hadoop-1.1.2]$ ./bin/hadoop fs -ls out
Found 3 items
-rw-r--r-- 1 xiaoyu supergroup 0 2013-09-10 14:20 /user/xiaoyu/out/_SUCCESS
drwxr-xr-x - xiaoyu supergroup 0 2013-09-10 14:19 /user/xiaoyu/out/_logs
-rw-r--r-- 1 xiaoyu supergroup 38 2013-09-10 14:20 /user/xiaoyu/out/part-r-00000
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -cat out/part-r-00000
! 1
Hadoop 2
Hello, 1
你好, 1
！ 1
[xiaoyu@hadoop02 hadoop-1.1.2]$
7. 删除HDFS测试文件和目录
[xiaoyu@hadoop02 hadoop-1.1.2] $ ./bin/hadoop fs -rmr TEST out
Deleted hdfs://hadoop01:9000/user/xiaoyu/TEST
Deleted hdfs://hadoop01:9000/user/xiaoyu/out
[xiaoyu@hadoop02 hadoop-1.1.2]$
至此，测试结束。


可以登录官网API文档查看http://hadoop.apache.org/docs/r1.1.2/api/org/apache/hadoop/examples/package-summary.html

更多的hadoop示例程序


     本文转自xiaoyuwang 51CTO博客，原文链接：http://blog.51cto.com/wangxiaoyu/1293707，如需转载请自行联系原作者



原文链接：https://yq.aliyun.com/articles/447904
关注公众号

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。

持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

转载内容版权归作者及来源网站所有，本站原创内容转载请注明来源。

相关文章

    文章评论

    共有0条评论来说两句吧...

    文章二维码

    扫描即可查看该文章

    点击排行

    推荐阅读

    最新文章