![]()
打开微信扫一扫，关注微信公众号【数据与算法联盟】
转载请注明出处：http://blog.csdn.net/gamer_gyt
博主微博：http://weibo.com/234654758 （欢迎互撩）
Github：https://github.com/thinkgamer
====================================================================
原文链接：http://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
目录
--| 概述（Overview）
--| URI‘s
--| HTTP 请求（HTTP Requests）
----------| HTTP操作总结（summary of http operations）
----------| 安全性（Security）
----------| header支持（headers Supported）
--| HTTP 回应 （HTTP Responses）
----------| 压缩
----------| 响应格式
----------| 响应错误
----------| 响应例子
-------------------| 单资源的JSON响应
-------------------| 错误资源的JSON响应
--| 用法示例 （Sample usage）
1、概述：
Hadoop YARN的web服务 REST API是一组向集群，节点，应用程序和应用历史信息提供访问的URI资源，该URI资源被分组为基于返回的信息的类型的API。一些URI资源返回集合而另一些返回单个信息。
2、URI’s
对于基于REST的web服务的URI语法如下
http://{http address of service}/ws/{version}/{resourcepath}
语法元素介绍：
http address of servicel：该服务的http地址获取有关信息，目前支持的有ResourceManager和NodeManager，MapReduce应用程序和历史记录服务
version：API的版本，在hadoop2.7.2中api的版本号为v1
resourcepath：他定义了一个单例资源或资源集合路径
3、HTTP Requests（Http请求）
如果要调用REST API，你的应用程序应执行与资源相关联的URI的HTTP操作
HTTP操作总结：
针对HTTP请求，目前只支持GET，它根据索引信息，活的相应的资源
安全性：
web服务 REST API作为web UI方面具有相应级别的安全性，如果您的群集管理员就启用筛选器，你必须通过他们所指定的机制进行身份验证。
Header 支持：
* Accept
* Accept-Encoding
目前在header使用的唯一领域便是Accept和Accept-Encoding，Accept支持你接受响应类型的XML和JSON，Accept-Encoding仅支持接受ZIP的编码，如果指定了，这将返回Gzio压缩格式，否则输出未压缩，所有其他头字段被忽略。
4、HTTP Responses：
接下来描述一些语法和Web服务的API REST的HTTP响应的其他细节
压缩：
此版本支持gzip压缩，如果您指定在HTTP请求的Accept-Encoding头的gzip （接受编码：gzip ）
响应格式：
本版本的web服务的REST API支持JSON和XML格式的响应， JSON是默认的。如果要设置响应格式，可以在构造HTTP请求时的Header中指定。
在HTTP响应代码中，作为指定的响应包，可以包含代表资源和错误消息的数据，在成功的情况下，响应包是一种选定的格式，JSON或者XML，在错误的情况下，该响应包是根据要求的格式返回的任一JSON或者XML格式的，响应的内容类型报头包含所请求的格式，如果该应用程序请求课一个不支持的格式，响应状态代码为500。注意，这里并没有指定字段在响应包的响应顺序，可能会发生变化。此外，附加的字段可能被添加到一个响应包，因此，您的应用程序应该使用分析例程，可以从一个响应体以任意顺序提取数据。
响应错误：
调用一个HTTP请求之后，应用程序应检查响应状态代码来验证是否请求成功或检测到一些错误。如果响应状态码标示错误，响应包中包含的错误信息。第一个字段是异常类型，目前只返回的RemoteException。下表列出了的RemoteException错误消息中的项目：
| Item |
Data Type |
Description |
| exception |
String |
exception type |
| javaClassName |
String |
java class name of exception |
| message |
String |
Detailed message of exception |
响应例子：
(1)、单资源的JSON响应
HTTP Request: GET http://rmhost.domain:8088/ws/v1/cluster/app/application\_1324057493980\_0001
Response Status Line: HTTP/1.1 200 OK
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
返回的响应包:
{
app":
{
"id":"application_1324057493980_0001",
"user":"user1",
"name":"",
"queue":"default",
"state":"ACCEPTED",
"finalStatus":"UNDEFINED",
"progress":0,
"trackingUI":"UNASSIGNED",
"diagnostics":"",
"clusterId":1324057493980,
"startedTime":1324057495921,
"finishedTime":0,
"elapsedTime":2063,
"amContainerLogs":"http:\/\/amNM:2\/node\/containerlogs\/container_1324057493980_0001_01_000001",
"amHostHttpAddress":"amNM:2"
}
} (2)、错误响应
我们试着去响应一个不存在的任务
HTTP Request: GET http://rmhost.domain:8088/ws/v1/cluster/app/application\_1324057493980\_9999
Response Status Line: HTTP/1.1 404 Not Found
Response Header:
HTTP/1.1 404 Not Found
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
返回的响应包
{
"RemoteException" : {
"javaClassName" : "org.apache.hadoop.yarn.webapp.NotFoundException",
"exception" : "NotFoundException",
"message" : "java.lang.Exception: app with id: application_1324057493980_9999 not found"
}
}
5、用发示例
这里主要用的是linux下的curl命令来获取响应的header和response package信息
这里分为三种情况
(1)：获取正在执行的job信息
提交一个task：
hadoop jar hadoop-mapreduce-test.jar sleep -Dmapred.job.queue.name=a1 -m 1 -r 1 -rt 1200000 -mt 20
shell终端会打印相应的日志：
12/01/18 04:25:15 INFO mapred.ResourceMgrDelegate: Submitted application application_1326821518301_0010 to ResourceManager at host.domain.com/10.10.10.10:8032
12/01/18 04:25:15 INFO mapreduce.Job: Running job: job_1326821518301_0010
12/01/18 04:25:21 INFO mapred.ClientServiceDelegate: The url to track the job: host.domain.com:8088/proxy/application_1326821518301_0010/
12/01/18 04:25:22 INFO mapreduce.Job: Job job_1326821518301_0010 running in uber mode : false
12/01/18 04:25:22 INFO mapreduce.Job: map 0% reduce 0%
执行命令获取信息：
curl --compressed -H "Accept: application/json" -X GET "http://host.domain.com:8088/ws/v1/cluster/apps/application_1326821518301_0010"
输出：
{
"app" : {
"finishedTime" : 0,
"amContainerLogs" : "http://host.domain.com:8042/node/containerlogs/container_1326821518301_0010_01_000001",
"trackingUI" : "ApplicationMaster",
"state" : "RUNNING",
"user" : "user1",
"id" : "application_1326821518301_0010",
"clusterId" : 1326821518301,
"finalStatus" : "UNDEFINED",
"amHostHttpAddress" : "host.domain.com:8042",
"progress" : 82.44703,
"name" : "Sleep job",
"startedTime" : 1326860715335,
"elapsedTime" : 31814,
"diagnostics" : "",
"trackingUrl" : "http://host.domain.com:8088/proxy/application_1326821518301_0010/",
"queue" : "a1"
}
} 如果想获取更多的信息：
curl --compressed -H "Accept: application/json" -X GET "http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs"
输出：
{
"jobs" : {
"job" : [
{
"runningReduceAttempts" : 1,
"reduceProgress" : 72.104515,
"failedReduceAttempts" : 0,
"newMapAttempts" : 0,
"mapsRunning" : 0,
"state" : "RUNNING",
"successfulReduceAttempts" : 0,
"reducesRunning" : 1,
"acls" : [
{
"value" : " ",
"name" : "mapreduce.job.acl-modify-job"
},
{
"value" : " ",
"name" : "mapreduce.job.acl-view-job"
}
],
"reducesPending" : 0,
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326860720902,
"id" : "job_1326821518301_10_10",
"successfulMapAttempts" : 1,
"runningMapAttempts" : 0,
"newReduceAttempts" : 0,
"name" : "Sleep job",
"mapsPending" : 0,
"elapsedTime" : 64432,
"reducesCompleted" : 0,
"mapProgress" : 100,
"diagnostics" : "",
"failedMapAttempts" : 0,
"killedReduceAttempts" : 0,
"mapsTotal" : 1,
"uberized" : false,
"killedMapAttempts" : 0,
"finishTime" : 0
}
]
}
} 如果用户想获得上边列出作业ID job_1326821518301_10_10工作任务详细信息。
curl --compressed -H "Accept: application/json" -X GET "http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs/job_1326821518301_10_10/tasks"
输出：
{
"tasks" : {
"task" : [
{
"progress" : 100,
"elapsedTime" : 5059,
"state" : "SUCCEEDED",
"startTime" : 1326860725014,
"id" : "task_1326821518301_10_10_m_0",
"type" : "MAP",
"successfulAttempt" : "attempt_1326821518301_10_10_m_0_0",
"finishTime" : 1326860730073
},
{
"progress" : 72.104515,
"elapsedTime" : 0,
"state" : "RUNNING",
"startTime" : 1326860732984,
"id" : "task_1326821518301_10_10_r_0",
"type" : "REDUCE",
"successfulAttempt" : "",
"finishTime" : 0
}
]
}
}
(2)、Map任务已经完成，但是reduce任务仍然在运行
用户希望得到reduce任务task_1326821518301_10_10_r_0任务企图的信息，请注意Accept-header并不是真的要求在这里，因为JSON是默认的输出格式
curl --compressed -X GET "http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs/job_1326821518301_10_10/tasks/task_1326821518301_10_10_r_0/attempts"
输出：
{
"taskAttempts" : {
"taskAttempt" : [
{
"elapsedMergeTime" : 158,
"shuffleFinishTime" : 1326860735378,
"assignedContainerId" : "container_1326821518301_0010_01_000003",
"progress" : 72.104515,
"elapsedTime" : 0,
"state" : "RUNNING",
"elapsedShuffleTime" : 2394,
"mergeFinishTime" : 1326860735536,
"rack" : "/10.10.10.0",
"elapsedReduceTime" : 0,
"nodeHttpAddress" : "host.domain.com:8042",
"type" : "REDUCE",
"startTime" : 1326860732984,
"id" : "attempt_1326821518301_10_10_r_0_0",
"finishTime" : 0
}
]
}
} reduce仍在进行，用户希望看到计数器的相关信息
curl --compressed -H "Accept: application/json" -X GET "http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs/job_1326821518301_10_10/tasks/task_1326821518301_10_10_r_0/attempts/attempt_1326821518301_10_10_r_0_0/counters"
输出：
{
"JobTaskAttemptCounters" : {
"taskAttemptCounterGroup" : [
{
"counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter" : [
{
"value" : 4216,
"name" : "FILE_BYTES_READ"
},
{
"value" : 77151,
"name" : "FILE_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "FILE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_WRITE_OPS"
},
{
"value" : 0,
"name" : "HDFS_BYTES_READ"
},
{
"value" : 0,
"name" : "HDFS_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "HDFS_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_WRITE_OPS"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
"counter" : [
{
"value" : 0,
"name" : "COMBINE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "COMBINE_OUTPUT_RECORDS"
},
{
"value" : 1767,
"name" : "REDUCE_INPUT_GROUPS"
},
{
"value" : 25104,
"name" : "REDUCE_SHUFFLE_BYTES"
},
{
"value" : 1767,
"name" : "REDUCE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "REDUCE_OUTPUT_RECORDS"
},
{
"value" : 0,
"name" : "SPILLED_RECORDS"
},
{
"value" : 1,
"name" : "SHUFFLED_MAPS"
},
{
"value" : 0,
"name" : "FAILED_SHUFFLE"
},
{
"value" : 1,
"name" : "MERGED_MAP_OUTPUTS"
},
{
"value" : 50,
"name" : "GC_TIME_MILLIS"
},
{
"value" : 1580,
"name" : "CPU_MILLISECONDS"
},
{
"value" : 141320192,
"name" : "PHYSICAL_MEMORY_BYTES"
},
{
"value" : 1118552064,
"name" : "VIRTUAL_MEMORY_BYTES"
},
{
"value" : 73728000,
"name" : "COMMITTED_HEAP_BYTES"
}
]
},
{
"counterGroupName" : "Shuffle Errors",
"counter" : [
{
"value" : 0,
"name" : "BAD_ID"
},
{
"value" : 0,
"name" : "CONNECTION"
},
{
"value" : 0,
"name" : "IO_ERROR"
},
{
"value" : 0,
"name" : "WRONG_LENGTH"
},
{
"value" : 0,
"name" : "WRONG_MAP"
},
{
"value" : 0,
"name" : "WRONG_REDUCE"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
"counter" : [
{
"value" : 0,
"name" : "BYTES_WRITTEN"
}
]
}
],
"id" : "attempt_1326821518301_10_10_r_0_0"
}
}
(3)、job已经完成，用户希望看到该作业的历史服务最后信息
curl --compressed -X GET "http://host.domain.com:19888/ws/v1/history/mapreduce/jobs/job_1326821518301_10_10"
输出：
{
"job" : {
"avgReduceTime" : 1250784,
"failedReduceAttempts" : 0,
"state" : "SUCCEEDED",
"successfulReduceAttempts" : 1,
"acls" : [
{
"value" : " ",
"name" : "mapreduce.job.acl-modify-job"
},
{
"value" : " ",
"name" : "mapreduce.job.acl-view-job"
}
],
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326860720902,
"id" : "job_1326821518301_10_10",
"avgMapTime" : 5059,
"successfulMapAttempts" : 1,
"name" : "Sleep job",
"avgShuffleTime" : 2394,
"reducesCompleted" : 1,
"diagnostics" : "",
"failedMapAttempts" : 0,
"avgMergeTime" : 2552,
"killedReduceAttempts" : 0,
"mapsTotal" : 1,
"queue" : "a1",
"uberized" : false,
"killedMapAttempts" : 0,
"finishTime" : 1326861986164
}
} 用户想获得ResourceManager的最后信息
curl --compressed -H "Accept: application/json" -X GET "http://host.domain.com:8088/ws/v1/cluster/apps/application_1326821518301_0010"
输出：
{
"app" : {
"finishedTime" : 1326861991282,
"amContainerLogs" : "http://host.domain.com:8042/node/containerlogs/container_1326821518301_0010_01_000001",
"trackingUI" : "History",
"state" : "FINISHED",
"user" : "user1",
"id" : "application_1326821518301_0010",
"clusterId" : 1326821518301,
"finalStatus" : "SUCCEEDED",
"amHostHttpAddress" : "host.domain.com:8042",
"progress" : 100,
"name" : "Sleep job",
"startedTime" : 1326860715335,
"elapsedTime" : 1275947,
"diagnostics" : "",
"trackingUrl" : "http://host.domain.com:8088/proxy/application_1326821518301_0010/jobhistory/job/job_1326821518301_10_10",
"queue" : "a1"
}
}
