一.简述

基于对Yarn ResourceManager中运行程序的状态（RUNNING、KILLED、FAILED、FINISHED）以及ApplicationMaster中Application的Job执行时长超过批次时间的监控，来达到对Spark on Yarn程序的失败重启、超时重启等功能

二.Yarn主要的几类API

1.查询整个集群指标

GET http:// http address:port>/ws/v1/cluster/metrics

2.查询集群调度器详情

GET http:// http address:port>/ws/v1/cluster/scheduler

3.监控任务

curl 'http:// http address:port>/ws/v1/cluster/apps//state'
GET http:// http address:port>/ws/v1/cluster/apps//state

4.查看指定任务

GET http:// http address:port>/ws/v1/cluster/apps/

5.查看指定任务的详细信息

curl http:// http address:port>/proxy//ws/v2/mapreduce/info"

6.杀死任务

yarn application -kill application_id
curl -v -X PUT -d '{"state": "KILLED"}''http:// http address:port>/ws/v1/cluster/apps/'
PUT http:// http address:port>/ws/v1/cluster/apps//state

三.YarnMonitor

Ⅰ. Setup

1. install yarn-api-client

you mast install yarn-api-client when you use this yarn monitor

• python setup.py build
• python setup.py install

2. uninstall yarn-api-client

when you need uninstall this yarn-api-client model,use this

• pip list
• pip uninstall yarn-api-client

3. upate yarn-api-client

when you need update python model,you need unintall and update

• update yarn-api-client
• cp yarn-api-client/base.py base.py.bak

4. offline intall python model

when you need intall other python model,you can do this

• pip freeze > yarn.txt
• mkdir yarnpackage
• pip install --no-index --find-links=yarnpackage/ -r yarn.txt

Ⅱ. Command

you should modify the script permissions

• chmod 774 start_prmsbd.sh

Ⅲ. Crontab

configure crontab task

• crontab -l
• crontab -e

1.start yarn monitor

*/1 * * * * ../yarnmonitor/yarn-monitor/command/start_yarn_monitor.sh >> ../yarnmonitor/yarn-monitor/logs/yarn-corntab.log 2>&1 * * * * * sleep 60; ../yarnmonitor/yarn-monitor/command/start_yarn_monitor.sh

2.clear yarn monitor log

0 4 * * * ../yarn-monitor/command/clear_log_opm.sh >> ../yarn-monitor/logs/yarn-corntab.log 2>&1 0 4 * * * ../yarnmonitor/yarn-monitor/command/clear_log_opm.sh >> ../yarnmonitor/yarn-monitor/logs/yarn-corntab.log 2>&1

Ⅳ. Just for test

1. start yarn command

../python ./yarn-monitor/YarnMonitor.py

2. Application_Master API

curl --compressed -H "Accept: application/json" -X GET "http://***:8088/ws/v1/cluster/apps" curl --compressed -H "Accept: application/json" -X GET "http://***:8088/ws/v1/cluster/apps" curl --compressed -H "Accept: application/json" -X GET "http://***:8088/proxy/application_1549963435527_0001/ws/v1/mapreduce/info" curl --compressed -H "Accept: application/json" -X GET "http://***:8088/proxy/application_1535085750394_0017/ws/v1/mapreduce/info" curl --compressed -H "Accept: application/json" -X GET "http://***:8088/proxy/application_1535085750394_0017/ws/v2/mapreduce/info" curl --compressed -H "Accept: application/json" -X GET "http://***:8088/proxy/application_1535085750394_0017/ws/v1/mapreduce/jobs/4536" curl --compressed -H "Accept: application/json" -X GET "http://***:8088/proxy/application_1548125170651_0090/api/v1/applications"

四.问题

其中，在ApplicationMaster中查询Job的返回数据无法转json的异常时，需修改yarn-api-client中修改对应API返回数据,可参考：

if 'ws/v1/mapreduce/info' in path: if response.status == OK: html_content = response.read() element_html = etree.HTML(html_content) tr_list = element_html.xpath('//tbody/tr') content_list = [] for tr in tr_list: item = {} item['id'] = tr.xpath('./td[1]/text()')[0].replace('\n', '').strip() item['duration'] = tr.xpath('./td[4]/text()')[0] # 打印每条信息 # logging.info(item) content_list.append(item) # print content_list return content_list response.close() return self.response_class(content_list) else: msg = 'Response finished with status: %s' % response.status raise APIError(msg)