Hadoop2.7实战v1.0之添加DataNode节点后,更改文件复制策略dfs.replication

2016-03-05 711 收藏

1.查看当前系统的复制策略dfs.replication为3,表示文件会备份成3份
a.通过查看hdfs-site.xml 文件

点击(此处)折叠或打开

1. [root@sht-sgmhadoopnn-01 ~]# cd /hadoop/hadoop-2.7.2/etc/hadoop
   
2. [root@sht-sgmhadoopnn-01 hadoop]# more hdfs-site.xml
   
3.  <property>
   
4.                 <name>dfs.replication</name>
   
5.                 <value>3</value>
   
6. </property>

b.通过查看当前hdfs文件的复制值是多少

点击(此处)折叠或打开

1. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -ls /testdir
   
2. Found 7 items
   
3. -rw-r--r-- 3 root supergroup 37322672 2016-03-05 17:59 /testdir/012_HDFS.avi
   
4. -rw-r--r-- 3 root supergroup 224001146 2016-03-05 18:01 /testdir/016_Hadoop.avi
   
5. -rw-r--r-- 3 root supergroup 176633760 2016-03-05 19:11 /testdir/022.avi
   
6. -rw-r--r-- 3 root supergroup 30 2016-02-28 22:42 /testdir/1.log
   
7. -rw-r--r-- 3 root supergroup 196 2016-02-28 22:23 /testdir/full_backup.log
   
8. -rw-r--r-- 3 root supergroup 142039186 2016-03-05 17:55 /testdir/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
   
9. -rw-r--r-- 3 root supergroup 44 2016-02-28 19:40 /testdir/test.log
   
10. [root@sht-sgmhadoopnn-01 hadoop]#
11. ### 紧跟-rw-r--r--权限后面的3,表示该文件在hdfs有多少份备份

c.通过 hadoop fsck /,也可以方便的看到Average block replication的值仍然为3，该值我们可以手动的进行动态修改。
而Default replication factor则需要重启整个Hadoop集群才能修改(就是hdfs-site.xml 文件中改为4,然后集群重启才生效,不过这种情况不适用生产集群),
但实际影响系统的还是Average block replication的值，因此并非一定要修改默认值Default replication factor。

点击(此处)折叠或打开

1. [root@sht-sgmhadoopnn-01 hadoop]# hdfs fsck /
   
2. 16/03/06 17:15:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   
3. Connecting to namenode via http://sht-sgmhadoopnn-01:50070/fsck?ugi=root&path=%2F
   
4. FSCK started by root (auth:SIMPLE) from /172.16.101.55 for path / at Sun Mar 06 17:15:29 CST 2016
   
5. ............Status: HEALTHY
   
6.  Total size: 580151839 B
   
7.  Total dirs: 15
   
8.  Total files: 12
   
9.  Total symlinks: 0
   
10.  Total blocks (validated): 11 (avg. block size 52741076 B)
    
11.  Minimally replicated blocks: 11 (100.0 %)
    
12.  Over-replicated blocks: 0 (0.0 %)
    
13.  Under-replicated blocks: 0 (0.0 %)
    
14.  Mis-replicated blocks: 0 (0.0 %)
    
15.  Default replication factor: 3
    
16.  Average block replication: 3.0
    
17.  Corrupt blocks: 0
    
18.  Missing replicas: 0 (0.0 %)
    
19.  Number of data-nodes: 4
    
20.  Number of racks: 1
    
21. FSCK ended at Sun Mar 06 17:15:29 CST 2016 in 9 milliseconds
    
22. The filesystem under path '/' is HEALTHY
    
23. You have mail in /var/spool/mail/root

2.修改hdfs文件备份系数

点击(此处)折叠或打开

1. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -help
   
2. -setrep [-R] [-w] ... :
   
3.   Set the replication level of a file. If is a directory then the command
   
4.   recursively changes the replication factor of all files under the directory tree
   
5.   rooted at .
   
6.                                                                                  
   
7.   -w It requests that the command waits for the replication to complete. This
   
8.       can potentially take a very long time.
   
9.   -R It is accepted for backwards compatibility. It has no effect.
   
10. 
    
11. 
    
12. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -setrep -w 4 -R /
    
13. setrep: `-R': No such file or directory
    
14. Replication 4 set: /out1/_SUCCESS
    
15. Replication 4 set: /out1/part-r-00000
    
16. Replication 4 set: /testdir/012_HDFS.avi
    
17. Replication 4 set: /testdir/016_Hadoop.avi
    
18. Replication 4 set: /testdir/022.avi
    
19. Replication 4 set: /testdir/1.log
    
20. Replication 4 set: /testdir/full_backup.log
    
21. Replication 4 set: /testdir/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
    
22. Replication 4 set: /testdir/test.log
    
23. Replication 4 set: /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1456590271264_0002-1456659654297-root-word+count-1456659679606-1-1-SUCCEEDED-root.root-1456659662730.jhist
    
24. Replication 4 set: /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1456590271264_0002.summary
    
25. Replication 4 set: /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1456590271264_0002_conf.xml
    
26. Waiting for /out1/_SUCCESS ... done
    
27. Waiting for /out1/part-r-00000 .... done
    
28. Waiting for /testdir/012_HDFS.avi ... done
    
29. Waiting for /testdir/016_Hadoop.avi ... done
    
30. Waiting for /testdir/022.avi ... done
    
31. Waiting for /testdir/1.log ... done
    
32. Waiting for /testdir/full_backup.log ... done
    
33. Waiting for /testdir/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm ... done
    
34. Waiting for /testdir/test.log ... done
    
35. Waiting for /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1456590271264_0002-1456659654297-root-word+count-1456659679606-1-1-SUCCEEDED-root.root-1456659662730.jhist ... done
    
36. Waiting for /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1456590271264_0002.summary ... done
    
37. Waiting for /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1456590271264_0002_conf.xml ... done
    
38. [root@sht-sgmhadoopnn-01 hadoop]#
    
39. 
    
40. ##再次检查备份系统的情况, Average block replication为4
    
41. [root@sht-sgmhadoopnn-01 hadoop]# hdfs fsck /
    
42. 16/03/06 17:25:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    
43. Connecting to namenode via http://sht-sgmhadoopnn-01:50070/fsck?ugi=root&path=%2F
    
44. FSCK started by root (auth:SIMPLE) from /172.16.101.55 for path / at Sun Mar 06 17:25:51 CST 2016
    
45. ............Status: HEALTHY
    
46.  Total size: 580151839 B
    
47.  Total dirs: 15
    
48.  Total files: 12
    
49.  Total symlinks: 0
    
50.  Total blocks (validated): 11 (avg. block size 52741076 B)
    
51.  Minimally replicated blocks: 11 (100.0 %)
    
52.  Over-replicated blocks: 0 (0.0 %)
    
53.  Under-replicated blocks: 0 (0.0 %)
    
54.  Mis-replicated blocks: 0 (0.0 %)
    
55.  Default replication factor: 3
    
56.  Average block replication: 4.0
    
57.  Corrupt blocks: 0
    
58.  Missing replicas: 0 (0.0 %)
    
59.  Number of data-nodes: 4
    
60.  Number of racks: 1
    
61. FSCK ended at Sun Mar 06 17:25:51 CST 2016 in 6 milliseconds
    
62. The filesystem under path '/

3.测试

点击(此处)折叠或打开

1. [root@sht-sgmhadoopnn-01 hadoop]# vi /tmp/wjp.log
   
2. hello,i am
   
3. hadoop
   
4. hdfs
   
5. mapreduce
   
6. yarn
   
7. hive
   
8. zookeeper
   
9. 
   
10. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -put /tmp/wjp.log /testdir
    
11. 
    
12. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -ls /testdir
    
13. Found 8 items
    
14. -rw-r--r-- 4 root supergroup 37322672 2016-03-05 17:59 /testdir/012_HDFS.avi
    
15. -rw-r--r-- 4 root supergroup 224001146 2016-03-05 18:01 /testdir/016_Hadoop.avi
    
16. -rw-r--r-- 4 root supergroup 176633760 2016-03-05 19:11 /testdir/022.avi
    
17. -rw-r--r-- 4 root supergroup 30 2016-02-28 22:42 /testdir/1.log
    
18. -rw-r--r-- 4 root supergroup 196 2016-02-28 22:23 /testdir/full_backup.log
    
19. -rw-r--r-- 4 root supergroup 142039186 2016-03-05 17:55 /testdir/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
    
20. -rw-r--r-- 4 root supergroup 44 2016-02-28 19:40 /testdir/test.log
    
21. -rw-r--r-- 3 root supergroup 62 2016-03-06 17:30 /testdir/wjp.log
    
22. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -rm /testdir/wjp.log
    
23. 16/03/06 17:31:47 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes.
    
24. Moved: 'hdfs://mycluster/testdir/wjp.log' to trash at: hdfs://mycluster/user/root/.Trash/Current
    
25. [root@sht-sgmhadoopnn-01 hadoop]#
    
26. ### put的测试文件wjp.log的备份数还是3,于是我先把测试文件删除掉,去修改namenode节点的hdfs-site.xml的参数

4.修改namenode节点的hdfs-site.xml的参数

点击(此处)折叠或打开

1. [root@sht-sgmhadoopnn-01 hadoop]# vi hdfs-site.xml
   
2.          <property>
   
3.                 <name>dfs.replication</name>
   
4.                 <value>4</value>
   
5.          </property>
   
6. [root@sht-sgmhadoopnn-01 hadoop]# scp hdfs-site.xml root@sht-sgmhadoopnn-02:/hadoop/hadoop-2.7.2/etc/hadoop
7. ###假如集群中,配置了namenode HA,那么应该需要对另外一个standbyNamenode节点的文件要同步一直,无需也同步到datanode节点

5.再次测试
##先不重启试试看

点击(此处)折叠或打开

1. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -put /tmp/wjp.log /testdir
   
2. 16/03/06 17:36:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   
3. You have mail in /var/spool/mail/root
   
4. [root@sht-sgmhadoopnn-01 hadoop]# hdfs dfs -ls /testdir
   
5. 16/03/06 17:36:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   
6. Found 8 items
   
7. -rw-r--r-- 4 root supergroup 37322672 2016-03-05 17:59 /testdir/012_HDFS.avi
   
8. -rw-r--r-- 4 root supergroup 224001146 2016-03-05 18:01 /testdir/016_Hadoop.avi
   
9. -rw-r--r-- 4 root supergroup 176633760 2016-03-05 19:11 /testdir/022.avi
   
10. -rw-r--r-- 4 root supergroup 30 2016-02-28 22:42 /testdir/1.log
    
11. -rw-r--r-- 4 root supergroup 196 2016-02-28 22:23 /testdir/full_backup.log
    
12. -rw-r--r-- 4 root supergroup 142039186 2016-03-05 17:55 /testdir/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
    
13. -rw-r--r-- 4 root supergroup 44 2016-02-28 19:40 /testdir/test.log
    
14. -rw-r--r-- 4 root supergroup 62 2016-03-06 17:36 /testdir/wjp.log
    
15. 
    
16. [root@sht-sgmhadoopnn-01 hadoop]# hdfs fsck /
    
17. 16/03/06 21:49:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    
18. Connecting to namenode via http://sht-sgmhadoopnn-01:50070/fsck?ugi=root&path=%2F
    
19. FSCK started by root (auth:SIMPLE) from /172.16.101.55 for path / at Sun Mar 06 21:49:12 CST 2016
    
20. ...............Status: HEALTHY
    
21.  Total size: 580152025 B
    
22.  Total dirs: 17
    
23.  Total files: 15
    
24.  Total symlinks: 0
    
25.  Total blocks (validated): 14 (avg. block size 41439430 B)
    
26.  Minimally replicated blocks: 14 (100.0 %)
    
27.  Over-replicated blocks: 0 (0.0 %)
    
28.  Under-replicated blocks: 0 (0.0 %)
    
29.  Mis-replicated blocks: 0 (0.0 %)
    
30.  Default replication factor: 3
    
31.  Average block replication: 4.0
    
32.  Corrupt blocks: 0
    
33.  Missing replicas: 0 (0.0 %)
    
34.  Number of data-nodes: 4
    
35.  Number of racks: 1
    
36. FSCK ended at Sun Mar 06 21:49:12 CST 2016 in 8 milliseconds

##【事实证明】:无需重启集群或者namenode节点,是从刚才动态设置命令(hdfs dfs -setrep -w 4 -R /)的内存信息中读取的,
而不是从配置文件hdfs-site.xml文件中读取配置的,从而验证了上面这句话:
实际影响系统的还是Average block replication的值，因此并非一定要修改默认值Default replication factor。

总结命令:
hdfs fsck /
hdfs dfs -setrep -w 4 -R /

上一篇 下一篇

微信关注我们

原文链接：https://yq.aliyun.com/articles/578890

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

Hadoop2.7实战v1.0之动态添加、删除DataNode节点及复制策略导向

一.环境 IP Host 及安装软件 部署模块 进程 172.16.101.55 sht-sgmhadoopnn-01 hadoop NameNode ResourceManager NameNode DFSZKFailoverController ResourceManager 172.16.101.56 sht-sgmhadoopnn-02 hadoop NameNode ResourceManager NameNode DFSZKFailoverController ResourceManager 172.16.101.58 sht-sgmhadoopdn-01 hadoop、zookeeper DataNode NodeManager Zookeeper DataNode NodeManager JournalNode QuorumPeerMain 172.16.101.59 sht-sgmhadoopdn-02 Hadoop、zookeeper DataNode NodeManager Zookeeper DataNode NodeManager JournalNode Q...

2016-03-05

761

上一篇

Hadoop2.7实战v1.0之HDFS HA

HDFS HA实战v1.0 当前环境: hadoop+zookeeper(namenode,resourcemanager HA) namenode serviceId Init status sht-sgmhadoopnn-01 nn1 active sht-sgmhadoopnn-02 nn2 standby 参考: http://blog.csdn.net/u011414200/article/details/50336735 一.查看namenode是active还是standby 1.打开网页 2.查看zkfc日志 点击(此处)折叠或打开 [root@sht-sgmhadoopnn-01 logs]# more hadoop-root-zkfc-sht-sgmhadoopnn-01.telenav.cn.log ………………….. 2016-02-28 00:24:00,692 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at sht-sgmhadoopnn-01/172.16...

2016-03-06

791

下一篇

相关文章

发表评论

发表

资源下载

更多资源

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

时间: 2025-12-12 大小: 211.28KB 下载: 54次

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

时间: 2026-01-27 大小: 1MB 下载: 5次

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

时间: 2025-12-17 大小: 5MB 下载: 1次

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

时间: 2026-01-03 大小: 1.42GB 下载: 2次