Fair Scheduler到Capacity Scheduler 转换工具
o 一次查看多个节点
o 细粒度的锁
o 多个分配线程
o 吞吐量提高5-10倍
• 介绍Fair Scheduler-> Capacity Scheduler转换工具
• 描述其内部运作方式
• 解释命令行开关
• 提供有关如何使用该工具的示例
• 解释该工具的局限性,因为尚无法从Fair Scheduler到Capacity Scheduler 100%自动转换
• 谈论未来的调度
yarn fs2cs -y /path/to/yarn-site.xml [-f /path/to/fair-scheduler.xml] {-o /output/path/ | -p} [-t] [-s] [-d]-y /path/to/yarn-site.xml [-f /path/to/fair-scheduler.xml] {-o /output/path/ | -p} [-t] [-s] [-d]
yarn fs2cs --yarnsiteconfig /path/to/yarn-site.xml [--fsconfig /path/to/fair-scheduler.xml] {--output-directory /output/path/ | --print} [--no-terminal-rule-check] [--skip-verification] [--dry-run]--yarnsiteconfig /path/to/yarn-site.xml [--fsconfig /path/to/fair-scheduler.xml] {--output-directory /output/path/ | --print} [--no-terminal-rule-check] [--skip-verification] [--dry-run]
yarn fs2cs --yarnsiteconfig /home/hadoop/yarn-site.xml --fsconfig /home/hadoop/fair-scheduler.xml --output-directory /tmp--yarnsiteconfig /home/hadoop/yarn-site.xml --fsconfig /home/hadoop/fair-scheduler.xml --output-directory /tmp
<allocations>
<queue name="root"><queue name="root">
<weight>1.0</weight><weight>1.0</weight>
<schedulingPolicy>drf</schedulingPolicy><schedulingPolicy>drf</schedulingPolicy>
<queue name="default"><queue name="default">
<weight>1.0</weight><weight>1.0</weight>
<schedulingPolicy>drf</schedulingPolicy><schedulingPolicy>drf</schedulingPolicy>
</queue></queue>
<queue name="users" type="parent"><queue name="users" type="parent">
<maxChildResources>memory-mb=8192, vcores=1</maxChildResources><maxChildResources>memory-mb=8192, vcores=1</maxChildResources>
<weight>1.0</weight><weight>1.0</weight>
<schedulingPolicy>drf</schedulingPolicy><schedulingPolicy>drf</schedulingPolicy>
</queue></queue>
</queue></queue>
<queuePlacementPolicy><queuePlacementPolicy>
<rule name="specified" create="true"/><rule name="specified" create="true"/>
<rule name="nestedUserQueue" create="true"><rule name="nestedUserQueue" create="true">
<rule name="default" create="true" queue="users"/><rule name="default" create="true" queue="users"/>
</rule></rule>
<rule name="default"/><rule name="default"/>
</queuePlacementPolicy></queuePlacementPolicy>
</allocations>
yarn.scheduler.fair.allow-undeclared-pools = true.scheduler.fair.allow-undeclared-pools = true
yarn.scheduler.fair.user-as-default-queue = true.scheduler.fair.user-as-default-queue = true
yarn.scheduler.fair.preemption = false.scheduler.fair.preemption = false
yarn.scheduler.fair.preemption.cluster-utilization-threshold = 0.8.scheduler.fair.preemption.cluster-utilization-threshold = 0.8
yarn.scheduler.fair.sizebasedweight = false.scheduler.fair.sizebasedweight = false
yarn.scheduler.fair.assignmultiple = true.scheduler.fair.assignmultiple = true
yarn.scheduler.fair.dynamicmaxassign = true.scheduler.fair.dynamicmaxassign = true
yarn.scheduler.fair.maxassign = -1.scheduler.fair.maxassign = -1
yarn.scheduler.fair.continuous-scheduling-enabled = false.scheduler.fair.continuous-scheduling-enabled = false
yarn.scheduler.fair.locality-delay-node-ms = 2000.scheduler.fair.locality-delay-node-ms = 2000
~$ yarn fs2cs -y /home/examples/yarn-site.xml -f /home/examples/fair-scheduler.xml -o /tmp$ yarn fs2cs -y /home/examples/yarn-site.xml -f /home/examples/fair-scheduler.xml -o /tmp
2020-05-05 14:22:41,384 INFO [main] converter.FSConfigToCSConfigConverter (FSConfigToCSConfigConverter.java:prepareOutputFiles(138)) - Output directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
-05-05 14:22:41,384 INFO [main] converter.FSConfigToCSConfigConverter (FSConfigToCSConfigConverter.java:prepareOutputFiles(138)) - Output directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
2020-05-05 14:22:41,388 INFO [main] converter.FSConfigToCSConfigConverter (FSConfigToCSConfigConverter.java:loadConversionRules(177)) - Conversion rules file is not defined, using default conversion config!
-05-05 14:22:41,388 INFO [main] converter.FSConfigToCSConfigConverter (FSConfigToCSConfigConverter.java:loadConversionRules(177)) - Conversion rules file is not defined, using default conversion config!
[for brevity output trimmed for brevity ] output trimmed
2020-05-05 14:22:42,572 ERROR [main] converter.FSConfigToCSConfigConverterMain (MarkerIgnoringBase.java:error(159)) - -05-05 14:22:42,572 ERROR [main] converter.FSConfigToCSConfigConverterMain (MarkerIgnoringBase.java:error(159)) - Error while starting FS configuration conversion! while starting FS configuration conversion!
[for brevity output trimmed for brevity ] output trimmed
Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Rules after rule 2 in queue placement policy can never be reached
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:115)
[ ]
~$ yarn fs2cs -y /home/examples/yarn-site.xml -f /home/examples/fair-scheduler.xml -o /tmp --no-terminal-rule-check
2020-05-05 14:41:39,189 INFO [main] capacity.CapacityScheduler (CapacityScheduler.java:initScheduler(384)) - Initialized CapacityScheduler with calculator=class org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator, minimumAllocation=<<memory:1024, vCores:1>>, maximumAllocation=<<memory:8192, vCores:4>>, asynchronousScheduling=false, asyncScheduleInterval=5ms,multiNodePlacementEnabled=false
2020-05-05 14:41:39,190 INFO [main] converter.ConvertedConfigValidator (ConvertedConfigValidator.java:validateConvertedConfig(72)) - Capacity scheduler was successfully started
This time, the conversion succeeded!
2020-05-05 14:41:38,908 WARN [main] converter.FSConfigToCSConfigRuleHandler (ConversionOptions.java:handleWarning(48)) - Setting <userMaxAppsDefault> is not supported, ignoring conversion
2020-05-05 14:41:38,945 WARN [main] converter.FSConfigToCSConfigRuleHandler (ConversionOptions.java:handleWarning(48)) - Setting <maxChildResources> is not supported, ignoring conversion
yarn.scheduler.capacity.resource-calculator =
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled = true
yarn.resourcemanager.scheduler.class = org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
yarn.scheduler.capacity.root.users.maximum-capacity = 100
yarn.scheduler.capacity.root.default.capacity = 50.000
yarn.scheduler.capacity.root.default.ordering-policy = fair
yarn.scheduler.capacity.root.users.capacity = 50.000
yarn.scheduler.capacity.root.default.maximum-capacity = 100
yarn.scheduler.capacity.root.queues = default,users
yarn.scheduler.capacity.root.maximum-capacity = 100
yarn.scheduler.capacity.maximum-am-resource-percent = 0.5
yarn.scheduler.fair.preemption - true
yarn.scheduler.fair.sizebasedweight - true
yarn.scheduler.fair.continuous-scheduling-enabled - true
yarn.scheduler.capacity.resource-calculator =
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn.scheduler.capacity.schedule-asynchronously.scheduling-interval-ms = 5
yarn.scheduler.capacity.schedule-asynchronously.enable = true
yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval = 10000
yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill = 15000
yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled = true
yarn.resourcemanager.scheduler.class =
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
yarn.resourcemanager.scheduler.monitor.enable = true
yarn.scheduler.capacity.root.default.ordering-policy.fair.enable-size-based-weight = true
yarn.scheduler.capacity.root.users.ordering-policy.fair.enable-size-based-weight = true
yarn.scheduler.capacity.root.users.capacity = 50.000
yarn.scheduler.capacity.root.queues = default,users
yarn.scheduler.capacity.root.users.maximum-capacity = 100
yarn.scheduler.capacity.root.ordering-policy.fair.enable-size-based-weight = true
[is omitted because it’s the same as before ] rest
root.a = 3
root.b = 1
yarn.scheduler.capacity.root.a.capacity = 75.000
yarn.scheduler.capacity.root.a.maximum-capacity = 100.000
yarn.scheduler.capacity.root.b.capacity = 25.000
yarn.scheduler.capacity.root.b.maximum-capacity = 100.000
root = 1
root.users = 20
root.default = 10
root.users.alice = 3
root.users.bob = 1
yarn.scheduler.capacity.root.capacity = 100.000
yarn.scheduler.capacity.root.maximum-capacity = 100.000
yarn.scheduler.capacity.root.users.capacity = 66.667
yarn.scheduler.capacity.root.users.maximum-capacity = 100.000
yarn.scheduler.capacity.root.default.capacity = 33.333
yarn.scheduler.capacity.root.default.maximum-capacity = 100.000
yarn.scheduler.capacity.root.users.alice.capacity = 75.000
yarn.scheduler.capacity.root.users.alice.maximum-capacity = 100.000
yarn.scheduler.capacity.root.users.bob.capacity = 25.000
yarn.scheduler.capacity.root.users.bob.maximum-capacity = 100.000
• 每个用户的最大应用程序数
• <userMaxAppsDefault> –每个用户的默认最大应用程序
• <minResources> –队列的最小资源
• <maxResources> –队列的最大资源
• <maxChildResources> –动态创建的队列的最大资源
• 队列级别的DRF排序策略:在Capacity Scheduler中,DRF必须是全局的。在Fair Scheduler中,可以在DRF父项下使用常规的“ Fair”策略。
1) 在Capacity Scheduler(YARN-9936 )中将百分比向量作为资源处理。用户将不仅可以定义单个容量,还可以定义不同资源的多个值。
2) 手柄maxRunningApps 每用户userMaxAppsDefault (YARN-9930 )我们有“每用户最多的应用程序”设置,但它不是直接配置和繁琐,因为它的三个设置的组合。我们还必须注意不要破坏现有行为–如果超过了最大设置,Capacity Scheduler中的现有逻辑将拒绝提交应用程序,而在Fair Scheduler中,该应用程序始终被接受,并将在以后进行调度。
3) 处理minResources ,maxResources 和maxChildResources 这些在很大程度上取决于YARN-9936 。在Fair Scheduler中,用户可以通过多种方式(单个百分比,两个单独的百分比或绝对资源)表达这些设置。为了支持Capacity Scheduler中的类似设置,我们需要YARN-9936 。
4) 使映射规则的行为类似于Fair Scheduler中存在的实现。在“放置规则”部分中说明了如何评估映射规则。我们可能需要一种新的,可插入的方法–这样,我们就不会在已经非常复杂的现有代码库中引入回归。
5) 关于DRF和其他调度策略的改进(YARN-9892 )当前,我们有一个由属性yarn.scheduler.capacity.resource-calculator 定义的全局资源计算器。这在Fair Scheduler中更加细腻。
6) 关于整个转换过程的通用微调在Capacity Scheduler 中有一些属性,例如“ user-limit-factor”或“ minimum-user-limit-percent”。我们暂时不使用这些设置,但是事实证明,在某些配置中,它们被证明是有用的。
本文分享自微信公众号 - 大数据杂货铺(bigdataGrocery)。
如有侵权,请联系 support@oschina.cn 删除。
本文参与“OSC源创计划”,欢迎正在阅读的你也加入,一起分享。

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。
持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。
转载内容版权归作者及来源网站所有,本站原创内容转载请注明来源。
- 上一篇
连续 3 年最受欢迎:Rust,香!
云栖号资讯:【点击查看更多行业资讯】在这里您可以找到不同行业的第一手的上云资讯,还在等什么,快来! 阿里妹导读:我们在选择一种开发语言时会综合考量各方面的特性,根据实际的需求适当取舍。鱼和熊掌往往不可兼得,要想开发效率高,必然要牺牲性能和资源消耗,反之亦然。但是Rust却出其不意,令人眼前一亮!本文将从性能、内存安全、开发效率、跨平台性及生态等五个方面,对Rust这一编程语言进行一些科普性质的分享。 一 性能对比 不同的语言使用不同的内存管理方式,一些语言使用垃圾回收机制在运行时寻找不再被使用的内存并释放,典型的如Java、Golang。在另一些语言中,程序员必须亲自分配和释放内存,比如C/C++。Rust 则选择了第三种方式:内存被一个所有权系统管理,它拥有一系列的规则使编译器在编译时进行检查,任何所有权系统的功能都不会导致运行时开销。Rust 速度惊人且内存利用率极高,标准Rust性能与标准C++性能不相上下,某些场景下效率甚至高于C++。由于没有运行时和垃圾回收,它能够胜任对性能要求特别高的服务。网上已经有了很多关于Rust性能分析对比的文章,不过为了获得一手的资料,还是自己动手...
- 下一篇
Guns-Seperation v1.1 发布,前后端分离版快速开发平台
Guns分离版全新gitee地址:https://gitee.com/stylefeng/guns-separation v1.1更新说明: 增加上传图片的预览功能 完善数据范围分配时候的判断逻辑 授权数据取消父级子级关联 【前端】工作台界面使用静态数据、环境显示抽屉默认设置为全显示 统一日志打印格式 修复邮件发送异常的问题 修复菜单遍历没有修改子应用的问题 默认去掉oss,cos,短信的依赖包,减少了默认打包体积 【pr合并】修改密码加密方式为bcrypt 修复定位bug 在线演示 账号密码:superAdmin/123456,地址:https://sep.stylefeng.cn 框架优势 模块化架构设计,层次清晰,业务层推荐写到单独模块,方便升级。 前后端分离架构,分离开发,分离部署,前后端互不影响。 前端技术采用vue + antdv + axios。 后端采用spring boot + mybatis-plus + hutool等,开源可靠。 基于spring security(jwt) + 用户UUID双重认证。 基于AOP实现的接口粒度的鉴权,最细粒度过滤权限资源。 基于...
相关文章
文章评论
共有0条评论来说两句吧...
文章二维码
点击排行
推荐阅读
最新文章
- Hadoop3单机部署,实现最简伪集群
- CentOS7编译安装Cmake3.16.3,解决mysql等软件编译问题
- SpringBoot2整合MyBatis,连接MySql数据库做增删改查操作
- Windows10,CentOS7,CentOS8安装MongoDB4.0.16
- CentOS6,CentOS7官方镜像安装Oracle11G
- CentOS8安装MyCat,轻松搞定数据库的读写分离、垂直分库、水平分库
- Docker快速安装Oracle11G,搭建oracle11g学习环境
- SpringBoot2整合Thymeleaf,官方推荐html解决方案
- SpringBoot2更换Tomcat为Jetty,小型站点的福音
- MySQL8.0.19开启GTID主从同步CentOS8