cdh4.6.0到cdh5.2.0 upgrade和rollback问题小结
upgrade小结：
1.dfsadmin -upgradeProgress status 在cdh5.2.0中没有，在4.6.0有（见源码org.apache.hadoop.hdfs.tools.DFSAdmin）
升级的时候不能通过这个看到升级状态
rollingUpgrade这个参数在4.6.0中不存在，在5.2.0中有，可以用于滚动升级
2.在cdh5.2.0中执行upgrade,nn中调用的命令是
|
1
|
hadoop-daemon.sh start namenode -upgrade
|
最终调用org.apache.hadoop.hdfs.server.namenode.NameNode类，并传入upgrade参数
3.yarn的设置有些变动，下面两个参数会影响nm是否启动正常
|
1
2
3
4
5
6
7
8
|
<property>
<name>yarn.nodemanager.aux-services<
/name
>
<value>mapreduce.shuffle<
/value
>
<
/property
>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class<
/name
>
<value>org.apache.hadoop.mapred.ShuffleHandler<
/value
>
<
/property
>
|
修改为：
|
1
2
3
4
5
6
7
8
|
<property>
<name>yarn.nodemanager.aux-services<
/name
>
<value>mapreduce_shuffle<
/value
>
<
/property
>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class<
/name
>
<value>org.apache.hadoop.mapred.ShuffleHandler<
/value
>
<
/property
>
|
4.impala1.1.1和cdh5.2.0有兼容性问题
cdh5.x开始使用PB做通信，需要升级impala到2.0.0（impala的可用性/稳定性/性能需要测试）
rollback小结：
1.rollback要在4.6.0版本上运行
在4.6.0上运行rollback,finalize,upgrade时，都会判断时否为ha模式，如果没有关闭ha的配置会报如下错误：
|
1
2
3
4
|
14
/11/19
15:25:47 FATAL namenode.NameNode: Exception
in
namenode
join
org.apache.hadoop.HadoopIllegalArgumentException: Invalid startup option. Cannot perform DFS upgrade with HA enabled.
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1130)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1241)
|
namenode类main方法中通过createNameNode方法创建一个NameNode的实例
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
public
static
void
main(String argv[])
throws
Exception {
if
(DFSUtil.parseHelpArgument(argv, NameNode.USAGE, System.out,
true
)) {
System.exit(
0
);
}
try
{
StringUtils.startupShutdownMessage(NameNode.
class
, argv, LOG);
NameNode namenode = createNameNode(argv,
null
);
if
(namenode !=
null
)
namenode.join();
}
catch
(Throwable e) {
LOG.fatal(
"Exception in namenode join"
, e);
terminate(
1
, e);
}
}
|
而在createNameNode方法中会通过下面的代码检测是否为ha的配置，而在5.2.0是没有这个限制的。
|
1
2
3
4
5
6
7
|
if
(HAUtil.isHAEnabled(conf, DFSUtil.getNamenodeNameServiceId(conf)) &&
(startOpt == StartupOption.UPGRADE ||
startOpt == StartupOption.ROLLBACK ||
startOpt == StartupOption.FINALIZE)) {
throw
new
HadoopIllegalArgumentException(
"Invalid startup option. "
+
"Cannot perform DFS upgrade with HA enabled."
);
}
|
这里会涉及到两个判断方法：
1）
|
1
2
3
4
5
6
7
8
|
org.apache.hadoop.hdfs.HAUti类的isHAEnabled方法：
public
static
boolean
isHAEnabled(Configuration conf, String nsId) {
Map<String, Map<String, InetSocketAddress>> addresses =
DFSUtil.getHaNnRpcAddresses(conf);
if
(addresses ==
null
)
return
false
;
Map<String, InetSocketAddress> nnMap = addresses.get(nsId);
return
nnMap !=
null
&& nnMap.size() >
1
;
}
|
这里会依次调用org.apache.hadoop.hdfs.DFSUtil类的
|
1
|
getHaNnRpcAddresses/getAddresses/getNameServiceIds/getAddressesForNameserviceId/getNameNodeIds
|
方法，解析dfs.nameservices/dfs.ha.namenodes.xxxx/dfs.namenode.rpc-address.xxxx的设置来获取每个nameserviceid到对应的nn rpc地址的对应关系（
|
1
|
Map<String, Map<String, InetSocketAddress>>
|
）并判断map value的size(如果dfs.ha.namenodes.x设置超过1个就算ha)，这里只要把配置改成下面即可
|
1
2
3
4
|
<property>
<name>dfs.ha.namenodes.bipcluster<
/name
>
<value>nn1<
/value
>
<
/property
>
|
2）如果设置了jn会报如下错误：
|
1
2
3
4
5
6
7
8
9
10
11
|
14
/11/19
16:47:32 FATAL namenode.NameNode: Exception
in
namenode
join
java.io.IOException: Invalid configuration: a shared edits
dir
must not be specified
if
HA is not enabled.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:576)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:513)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:621)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:606)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1177)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1241)
14
/11/19
16:47:32 INFO util.ExitUtil: Exiting with status 1
|
错误在FSNamesystem类的构造方法中:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
final
boolean
persistBlocks = conf.getBoolean(DFS_PERSIST_BLOCKS_KEY,
DFS_PERSIST_BLOCKS_DEFAULT);
//默认是false
// block allocation has to be persisted in HA using a shared edits directory
// so that the standby has up-to-date namespace information
String nameserviceId = DFSUtil.getNamenodeNameServiceId(conf);
this
.haEnabled = HAUtil.isHAEnabled(conf, nameserviceId);
this
.persistBlocks = persistBlocks || (haEnabled && HAUtil.usesSharedEditsDir(conf));
// Sanity check the HA-related config.
if
(nameserviceId !=
null
) {
LOG.info(
"Determined nameservice ID: "
+ nameserviceId);
//Determined nameservice ID: bipcluster
}
LOG.info(
"HA Enabled: "
+ haEnabled);
//HA Enabled: false
if
(!haEnabled && HAUtil.usesSharedEditsDir(conf)) {
//异常由这里抛出
LOG.warn(
"Configured NNs:\n"
+ DFSUtil.nnAddressesAsString(conf));
//Configured NNs: Nameservice <bipcluster>: NN ID nn1 => xxxx:8020
throw
new
IOException(
"Invalid configuration: a shared edits dir "
+
"must not be specified if HA is not enabled."
);
}
|
HAUtil.usesSharedEditsDir方法：
|
1
2
3
|
public
static
boolean
usesSharedEditsDir(Configuration conf) {
return
null
!= conf.get(DFS_NAMENODE_SHARED_EDITS_DIR_KEY);
}
|
判断jn的edit dir设置，如果设置了dfs.namenode.shared.edits.dir就会抛出异常
去掉下面的设置即可：
|
1
2
3
4
|
<
property
>
<
name
>dfs.namenode.shared.edits.dir</
name
>
<
value
>qjournal://xxxxx/bipcluster</
value
>
</
property
>
|
2.rollback的时候要先把ha的配置更改为非ha的配置，然后进行rollback
rollback之后，再重做ha
重做ha的步骤简单如下：
1).关闭整个集群(非ha的),更改配置为ha的配置，备份原来的standby nn和jn的数据目录
2).删除旧的jn数据，并单独启动jn
|
1
|
.
/hadoop-daemon
.sh start journalnode
|
3).在active nn上运行
|
1
|
hdfs namenode -initializeSharedEdits
|
namenode通过initializeSharedEdits命令来初始化journalnode，把edits文件共享到journalnode上
4).运行active nn
|
1
|
.
/hadoop-daemon
.sh start namenode
|
5).在standby的nn上运行
|
1
2
|
hadoop-daemon.sh start namenode -bootstrapStandby
hadoop-daemon.sh start namenode
|
同步元数据并启动standby namenode
6).启动所有的dn
|
1
|
.
/hadoop-daemons
.sh start datanode
|
7).nn transitionToActive
|
1
2
|
hdfs haadmin -transitionToActive nn1
hdfs haadmin -getServiceState nn1
|
3.start-dfs.sh 这个脚本有bug，在传入-rollback时，只能dn可以rollback,nn不能rollback
|
1
2
3
4
5
|
diff
../..
/hadoop-2
.5.0-cdh5.2.0
/sbin/start-dfs
.sh start-dfs.sh
50c50
< nameStartOpt=
"$nameStartOpt $@"
---
> nameStartOpt=
"$nameStartOpts $@"
|
也可以直接通过
|
1
|
sh -x .
/hadoop-daemon
.sh start namenode -rollback
|
命令rollback nn，注意dn还是要rollback的
4.如果rollback或upgrade失败，可以通过之前的备份元数据进行覆盖恢复
本文转自菜菜光 51CTO博客，原文链接：http://blog.51cto.com/caiguangguang/1579873，如需转载请自行联系原作者
