使用自定义指标进行Pod弹性伸缩

Horizontal Pod Autoscaling，简称HPA，是Kubernetes中实现Pod水平自动伸缩功能的资源对象。Kubernetes 1.9 或更高版本支持HPA V2版本，允许我们是用自定义监控指标来控制Pod数目。
本文用阿里云容器服务来介绍HPA使用自定义指标进行弹性伸缩的功能。

安装配置Prometheus和Prometheus adaptor

我们使用Prometheus收集指标，通过Prometheus adaptor来扩展Kubernetes 自定义监控指标。
使用阿里云容器服务控制台创建好集群，并创建monitoring命名空间。然后进行如下步骤：

安装Prometheus

在应用目录-App Hub中搜索prometheus可以看到如下组件

点击prometheus进入安装页面，点击参数进行查看并配置（本文persistentVolume enabled设置为false，即使用emptyDir），选择集群命名空间后点击安装。

安装Prometheus adaptor

安装过程同上，注意配置参数中prometheus url要使用之前部署好的prometheus地址

验证安装是否成功

使用控制台的cloudshell或kubectl连接集群，执行如下命令：

kubectl api-versions

结果中可以看到如下内容，表明集群支持hpa v2并可以使用自定义监控指标

autoscaling/v2beta1 custom.metrics.k8s.io/v1beta1

我们可以进一步查看监控项，如pod支持的监控指标

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" |jq . | grep "pods/"

 "name": "pods/start_time_seconds", "name": "pods/cpu_user", "name": "pods/fs_limit_bytes", "name": "pods/cpu_cfs_throttled", "name": "pods/memory_failcnt", "name": "pods/fs_writes_bytes", "name": "pods/kube_pod_created", "name": "pods/kube_pod_status_phase", "name": "pods/tasks_state", "name": "pods/last_seen", "name": "pods/kube_pod_container_status_running", "name": "pods/fs_reads_merged", "name": "pods/memory_swap", "name": "pods/spec_cpu_quota", "name": "pods/fs_io_current", "name": "pods/kube_pod_container_resource_requests_memory_bytes", "name": "pods/kube_pod_container_status_waiting", "name": "pods/fs_sector_reads", "name": "pods/kube_pod_status_scheduled", "name": "pods/kube_pod_container_status_waiting_reason", "name": "pods/fs_reads_bytes", "name": "pods/kube_pod_container_resource_requests_cpu_cores", "name": "pods/kube_pod_container_status_terminated_reason", "name": "pods/cpu_system", "name": "pods/fs_read", "name": "pods/fs_writes", "name": "pods/kube_pod_owner", "name": "pods/memory_mapped_file", "name": "pods/spec_cpu_shares", "name": "pods/kube_pod_container_resource_limits_cpu_cores", "name": "pods/cpu_usage", "name": "pods/kube_pod_container_resource_limits", "name": "pods/kube_pod_info", "name": "pods/memory_working_set_bytes", "name": "pods/fs_writes_merged", "name": "pods/memory_usage_bytes", "name": "pods/kube_pod_container_resource_requests", "name": "pods/fs_io_time_weighted", "name": "pods/spec_cpu_period", "name": "pods/spec_memory_reservation_limit_bytes", "name": "pods/cpu_cfs_throttled_periods", "name": "pods/kube_pod_container_resource_limits_memory_bytes", "name": "pods/cpu_cfs_periods", "name": "pods/kube_pod_status_ready", "name": "pods/kube_pod_container_status_terminated", "name": "pods/memory_failures", "name": "pods/memory_rss", "name": "pods/spec_memory_swap_limit_bytes", "name": "pods/memory_cache", "name": "pods/kube_pod_start_time", "name": "pods/fs_io_time", "name": "pods/fs_reads", "name": "pods/fs_inodes_free", "name": "pods/kube_pod_container_status_ready", "name": "pods/fs_sector_writes", "name": "pods/cpu_load_average_10s", "name": "pods/spec_memory_limit_bytes", "name": "pods/kube_pod_labels", "name": "pods/kube_pod_container_status_restarts", "name": "pods/kube_pod_container_info", "name": "pods/fs_usage_bytes", "name": "pods/kube_pod_completion_time", "name": "pods/kube_pod_container_status_last_terminated_reason", "name": "pods/kube_pod_status_scheduled_time", "name": "pods/memory_max_usage_bytes", "name": "pods/fs_inodes", "name": "pods/fs_write",

部署Deployment

在容器服务控制台中点击无状态->使用镜像创建，创建并部署一个nginx deployment，配置如下

在访问设置中配置服务

创建成功后可以在无状态列表中找到名为nginx的deployment，点击详情可以查看到相关信息

创建HPA

在上述详情页中点击容器组水平伸缩器，点击创建，配置相关内容（指标使用pods，参考上述自定义监控指标内容填写一项，这里使用memory_usage_bytes，阈值10485760=10M）

创建成功后可以在列表中查看到nginx-hpa

测试并触发自动伸缩

使用压测工具ab进行压测

for a in `seq 1 50`; do ab -rSqd -c 100 -n 2000 http://ng.cfc7d1315bdd14d529a7f102c2163736d.cn-beijing.alicontainer.com/;done

等待一段时间，可以看到pod数量的变化

总结

通过 Prometheus 来监控应用程序并暴露出用于弹性伸缩的指标，使用HPA利用这些监控指标对应用进行弹性伸缩，可以应对一些突发状况，提高应用的可用性。