Services and State with Ambari REST API-低调大师

Services and State with Ambari REST API

2017-11-14 504

The Ambari management tool for Hadoop offers among other handy tools a comprehensive REST API for cluster administration. Logically a cluster is divided into hosts, services and service components. While the UI might not always has support for all needed scenarios sure the REST API can be used to achieve it. For example moving a master component of a service from one host to another.

In this post we are going to look a little closer at the way the Ambari API can be used to manage Hadoop services. At the end of this post you will find a list of all the currently supported Hadoop services with all the needed master, slave and client components that can be manged and administrated within your HDP stack. Also this posts contains the possible states and state transitions a component might have which could become useful when facing problems like Host config is in invalid state.

The possible states a service component can be in is listed below. To change or get the state of a service component the following API calls can be issued.

 
         // Getting the state of the component
        
         curl -k -u admin:admin -H 
         "X-Requested-By:ambari" 
         -i -X GET 
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/YARN_CLIENT 
        
         // Setting the state of the component to INSTALLED
        
         curl -k -u admin:admin -H 
         "X-Requested-By:ambari" 
         -i -X PUT  
        
         -d '{
         "RequestInfo"
         :{
         "context"
         :
         "Install YARN_CLIENT"
         }, 
        
         "Body"
         :{
         "HostRoles"
         :{
         "state"
         :
         "INSTALLED"
         }}}'  
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/YARN_CLIENT

Among the need to change the state of a service component a very common issue is the need to actually move one service – most likely a master service – to different host. Users faced with this problems often need to issue the existing API due to a lack of support by the existing UI – at least with older releases of Ambari Web UI. This is an example of moving the OOZIE_SERVER to a new host.

 
         // Stop the old component by putting it back in state INSTALLED
        
         curl -u admin:admin -H 
         "X-Requested-By:ambari" 
         -i -X PUT  
        
         -d 
         '{"RequestInfo": {"context": "Stop Oozie","query":"HostRoles/component_name.in('
         OOZIE_SERVER')"},  
        
         "Body"
         :{
         "HostRoles"
         : {
         "state"
         : 
         "INSTALLED"
         }}}'  
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts
        
         /<OLD_HOST_FQDN>/host_components/OOZIE_SERVER
        
         // Add component to host
        
         curl -u admin:admin -H 
         "X-Requested-By:ambari" 
         -i -X POST -d  
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts
        
         /<NEW_HOST_FQDN>/host_components/OOZIE_SERVER
        
         // Install component by achieving stated INSTALLED
        
         curl -u admin:admin -H 
         "X-Requested-By:ambari" 
         -i -X PUT  
        
         -d 
         '{"RequestInfo": {"context": "Install Oozie","query":"HostRoles/component_name.in('
         OOZIE_SERVER')"},  
        
         "Body"
         :{
         "HostRoles"
         : {
         "state"
         : 
         "INSTALLED"
         }}}'  
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts
        
         /<NEW_HOST_FQDN>/host_components/OOZIE_SERVER
        
         // Delete old component
        
         curl -u admin:admin -H 
         "X-Requested-By:ambari" 
         -i -X DELETE  
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts
        
         /<OLD_HOST_FQDN>/host_components/OOZIE_SERVER
        
         // Starting the new component
        
         curl -u admin:admin -H 
         "X-Requested-By:ambari" 
         -i -X PUT  
        
         -d 
         '{"RequestInfo": {"context": "Start Oozie","query":"HostRoles/component_name.in('
         OOZIE_SERVER')"},  
        
         "Body"
         :{
         "HostRoles"
         : {
         "state"
         : 
         "STARTED"
         }}}'  
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts
        
         /<NEW_HOST_FQDN>/host_components/OOZIE_SERVER

Removing a complete service from the cluster would require all the removal of all it’s components together with the service itself. For example for removing the Knox service we would need to remove the KNOX_GATEWAY component from the cluster, but before let’s get their state:

 
         // Get service info
        
         curl -u admin:admin -H 
         "X-Requested-by:ambari" 
         -i -k -X GET 
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KNOX/
        
         // Stop component KNOX_GATEWAY
        
         curl -u admin:admin -H 
         "X-Requested-by:ambari" 
         -i -k -X PUT  
        
         -d 
         '{"ServiceComponentInfo": {"state": "INSTALLED"}}' 
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KNOX/components/KNOX_GATEWAY
        
         // Stop service KNOX
        
         curl -u admin:admin -H 
         "X-Requested-by:ambari" 
         -i -k -X PUT  
        
         -d 
         '{"ServiceInfo": {"state": "INSTALLED"}}' 
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KNOX
        
         // Delete component
        
         curl -u admin:admin -H 
         "X-Requested-by:ambari" 
         -i -k -X DELETE 
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/KNOX_GATEWAY
        
         // Delete service
        
         curl -u admin:admin -H 
         "X-Requested-by:ambari" 
         -i -k -X DELETE 
        
         http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KNOX

Possible states of a component:

StateDescription

INITInitial/Clean state.

INSTALLINGIn the process of installing.

INSTALL_FAILEDInstall failed.

INSTALLEDState when install completed successfully.

STARTINGIn the process of starting.

STARTEDState when start completed successfully.

STOPPINGIn the process of stopping.

UNINSTALLINGIn the process of uninstalling.

UNINSTALLEDState when uninstall completed successfully.

WIPING_OUTIn the process of wiping out the install.

UPGRADINGIn the process of upgrading the deployed bits.

DISABLEDDisabled master’s backup state.

UNKNOWNState could not be determined.

Allowed State transitions of a component:

Desired StateStart State

INSTALLED <-INIT, UNINSTALLED, INSTALLED, INSTALLING, STARTED, INSTALL_FAILED, UPGRADING, STOPPING, UNKNOWN, DISABLED

STARTED <-INSTALLED, STARTING, STARTED

UNINSTALLED <-INSTALLED, UNINSTALLED, UNINSTALLING

INIT <-UNINSTALLED, INIT, WIPING_OUT

DISABLED <-INSTALLED, INSTALL_FAILED, UNKNOWN

Services

A Hadoop cluster contains of multiple services that individually have either master, slave or client components. Below you will find a list of all currently supported components as part of a HDP stack divided in master, slave, or client groups. The cardinality notates the number of components that can exist at the same time in one cluster.

Master Services

NameAmbari Component Name

Service

Cardinality

NameNodeNAMENODEHDFS1-2

Secondary NameNodeSECONDARY_NAMENODEHDFS1

ResourceMangerRESOURCEMANAGERYARN1-2

Application Timeline ServerAPP_TIMELINE_SERVERYARN1

HistoryServerHISTORYSERVERMAPREDUCE2 1

Hive MetastoreHIVE_METASTOREHIVE1-2

HiveServer2HIVE_SERVERHIVE1-2

WebHcat ServerWEBHCAT_SERVERHIVE1

HBase MasterHBASE_MASTERHBASE1+

Spark Job History ServerSPARK_JOBHISTORYSERVERSPARK1

Nimbus ServerNIMBUSSTORM1

Storm REST ServerSTORM_REST_APISTORM1

Storm UISTORM_UI_SERVERSTORM1

DRPC ServerDRPC_SERVERSTORM1

Falcon ServerFALCON_SERVERFALCON1

ZookeeperZOOKEEPER_SERVERZOOKEEPER1+ (odd #)

Kafka BrokerKAFKA_BROKERKAFKA1+

Knox GatewayKNOX_GATEWAYKNOX1+

Ranger Admin ServerRANGER_ADMINRANGER1-3

Ranger User SyncRANGER_USERSYNCRANGER1

Ranger Key Management ServerRANGER_KMS_SERVERRANGER_KMS1+

Oozie ServerOOZIE_SERVEROOZIE1

Ganglia ServerGANGLIA_SERVERGANGLIA1

Nagios ServerNAGIOS_SERVERNAGIOS1

Ambari Metrics ServiceMETRICS_COLLECTORAMS1

Zeppelin ServerZEPPELIN_MASTERSPARK / HIVE1

Slave Services

NameAmbari Component Name

ServiceCardinality

DataNodeDATANODEHDFS1+

Journale Nodes for NameNode HAJOURNALNODEHDFS0+ (odd #)

Zookeeper Failover ServiceZKFCHDFS0+

Secondary NameNodeNFS_GATEWAYHDFS0+

Node ManagerNODEMANAGERYARN1+

HBase RegionServerHBASE_REGIONSERVERHBASE1+

Phoneix Query ServerPHOENIX_QUERY_SERVERHBASE0+

Storm SupervisorSUPERVISORSTORM1+

Ganglia Metrics CollectorGANGLIA_MONITORGANGLIAALL

Ambari Metrics CollectorMETRICS_MONITORAMSALL

Clients

NameAmbari Component Name

ServiceCardinality

HDFS ClientHDFS_CLIENTHDFS1+

YARN ClientYARN_CLIENTYARN1+

MapReduce ClientMAPREDUCE2_CLIENTMAPREDUCE21+

Spark ClientSPARK_CLIENTSPARK1+

Falcon ClientFALCON_CLIENTFALCON1+

HBase ClientHBASE_CLIENTHBASE1+

Hive ClientHIVE_CLIENTHIVE1+

HCat ClientHCATHIVE1+

Mahout ClientMAHOUTMAHOUT0+

Oozie ClientOOZIE_CLIENTOOZIE1+

Sqoop ClientSQOOPSQOOP1+

Zookeeper ClientZOOKEEPER_CLIENTZOOKEEPER1+

本文转自 yntmdr 51CTO博客，原文链接：http://blog.51cto.com/yntmdr/1775098，如需转载请自行联系原作者

微信关注我们

原文链接：https://yq.aliyun.com/articles/510973

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

hive 中的关系运算不等于

1.等值比较: = 语法：A=B 操作类型：所有基本类型描述: 如果表达式A与表达式B相等，则为TRUE；否则为FALSE 举例： hive>select 1 from lxw_dual where 1=1; 1 2.不等值比较: <> 语法: A <> B 操作类型: 所有基本类型描述: 如果表达式A为NULL，或者表达式B为NULL，返回NULL；如果表达式A与表达式B不相等，则为TRUE；否则为FALSE 举例： hive> select1 from lxw_dual where 1 <> 2; 1 注意：由于在通常得SQL写法中不等于也可以这样写 != 。但在hive中，当一个string类型和int类型比较的时候会出现问题。数字和数字类型用可以用！= 比较： hive>select*fromt_pvorderwhere1!=1andstart_date=20130306limit1; OK Timetaken:0.079seconds 带引号的数字和数字类型可以用！=比较： hive>sele...

2017-11-15

539

Pluggable Similarity Algorithms Before we move on from relevance and scoring, we will finish this chapter with a more advanced subject: pluggable similarity algorithms.While Elasticsearch uses theLucene’s Practical Scoring Functionas its default similarity algorithm, it supports other algorithms out of the box, which are listed in theSimilarity Modulesdocumentation. Okapi BM25 The most interesting competitor to TF/IDF and the vector space model is calledOkapi BM25, which is considered to be asta...

2017-11-15

699

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。