k8s原生的集群监控方案（Heapster+InfluxDB+Grafana)

1. Heapster+InfluxDB+Grafana简介
   heapster是一个监控计算、存储、网络等集群资源的工具，以k8s内置的cAdvisor作为数据源收集集群信息，并汇总出有价值的性能数据(Metrics)：cpu、内存、network、filesystem等，然后将这些数据输出到外部存储(backend)，如InfluxDB，最后再通过相应的UI界面进行可视化展示，如grafana。 另外heapster的数据源和外部存储都是可插拔的，所以可以很灵活的组建出很多监控方案，如：Heapster+ElasticSearch+Kibana等等。
2. Heapster的整体架构图
   
3. 创建InfluxDB资源对象

   #下载influxdb.yaml
   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
       name: monitoring-influxdb
       namespace: kube-system
   spec:
       replicas: 1
       template:
           metadata:
               labels:
                   task: monitoring
                   k8s-app: influxdb
           spec:
               containers:
               - name: influxdb
                   image: k8s.gcr.io/heapster-influxdb-amd64:v1.3.3
                   volumeMounts:
                   - mountPath: /data
                       name: influxdb-storage
               volumes:
               - name: influxdb-storage
                   emptyDir: {}
   ---
   apiVersion: v1
   kind: Service
   metadata:
       labels:
           task: monitoring
           #For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
           #If you are NOT using this as an addon, you should comment out this line.
   kubernetes.io/cluster-service: 'true'
   kubernetes.io/name: monitoring-influxdb
       name: monitoring-influxdb
       namespace: kube-system
   spec:
       type: NodePort
       ports:
       - nodePort: 31001
           port: 8086
           targetPort: 8086
       selector:
           k8s-app: influxdb

   所需的Heapster+InfluxDB+Grafana配置文件，请在Kubernetes Dashboard1.8.3部署中的yaml链接中下载使用。

   #influxdb.yaml文件需更改的地方：
   （1） image: k8s.gcr.io/heapster-influxdb-amd64:v1.3.3 （换成自己的images）
   ##说明：这里我在前文中提供的有images下载链接，直接下载使用不用更改！
   （2）这里我们使用NotePort暴露monitoring-influxdb服务在主机的31001端口上，那么InfluxDB服务端的地址：http://[host-ip]:31001 ，记下这个地址，以便创建heapster和为grafana配置数据源时，可以直接使用。

   spec:
       type: NodePort
       ports:
           - nodePort: 31001
               port: 8086
               targetPort: 8086
           selector:
               k8s-app: influxdb

4. 创建Grafana资源对象

   #下载grafana.yaml
   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
       name: monitoring-grafana
       namespace: kube-system
   spec:
       replicas: 1
       template:
           metadata:
               labels:
                   task: monitoring
                   k8s-app: grafana
           spec:
               containers:
               - name: grafana
                   image: k8s.gcr.io/heapster-grafana-amd64:v4.4.3
                   ports:
                   - containerPort: 3000
                       protocol: TCP
                   volumeMounts:
                   - mountPath: /etc/ssl/certs
                       name: ca-certificates
                       readOnly: true
                   - mountPath: /var
                       name: grafana-storage
                   env:
                   - name: INFLUXDB_HOST
                       value: monitoring-influxdb
                   - name: GF_SERVER_HTTP_PORT
                       value: "3000"
                       #The following env variables are required to make Grafana accessible via
                       #the kubernetes api-server proxy. On production clusters, we recommend
                       #removing these env variables, setup auth for grafana, and expose the grafana
                       #service using a LoadBalancer or a public IP.
                   - name: GF_AUTH_BASIC_ENABLED
                       value: "false"
                   - name: GF_AUTH_ANONYMOUS_ENABLED
                       value: "true"
                   - name: GF_AUTH_ANONYMOUS_ORG_ROLE
                       value: Admin
                   - name: GF_SERVER_ROOT_URL
                       #If you're only using the API Server proxy, set this value instead:
                       #value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
                       value: /
               volumes:
               - name: ca-certificates
                   hostPath:
                       path: /etc/ssl/certs
               - name: grafana-storage
                   emptyDir: {}
   ---
   apiVersion: v1
   kind: Service
   metadata:
       labels:
           #For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
           #If you are NOT using this as an addon, you should comment out this line.
           kubernetes.io/cluster-service: 'true'
           kubernetes.io/name: monitoring-grafana
       name: monitoring-grafana
       namespace: kube-system
   spec:
       #In a production setup, we recommend accessing Grafana through an external Loadbalancer
       #or through a public IP.
       #type: LoadBalancer
       #You could also use NodePort to expose the service at a randomly-generated port
       #type: NodePort
       type: NodePort
       ports:
       - nodePort: 30108
           port: 80
           targetPort: 3000
       selector:
           k8s-app: grafana

   ##说明

   虽然Heapster已经预先配置好了Grafana的Datasource和Dashboard，但是为了方便访问，这里我们使用NotePort暴露monitoring-grafana服务在主机的30108上，那么Grafana服务端的地址：http://192.168.245.16:30108 ，通过浏览器访问，为Grafana修改数据源，如下：
   
   标红的地方，为上一步记录下的InfluxDB服务端的地址。

5. 创建Heapster资源对象

   #下载heapster-rbac.yaml  

   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
       name: heapster
   roleRef:
       apiGroup: rbac.authorization.k8s.io
       kind: ClusterRole
       name: system:heapster
   subjects:
   - kind: ServiceAccount
       name: heapster
       namespace: kube-system

   #下载heapster.yaml  

   apiVersion: v1
   kind: ServiceAccount
   metadata:
       name: heapster
       namespace: kube-system
   ---
   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
       name: heapster
       namespace: kube-system
   spec:
       replicas: 1
       template:
           metadata:
               labels:
                   task: monitoring
                   k8s-app: heapster
           spec:
               serviceAccountName: heapster
               containers:
               - name: heapster
                   image: k8s.gcr.io/heapster-amd64:v1.5.3
                   imagePullPolicy: IfNotPresent
                   command:
                   - /heapster
                   - --source=kubernetes:https://kubernetes.default
               #- --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
                   - --sink=influxdb:http://192.168.246.167:31001 #influxdb服务端地址
   ---
   apiVersion: v1
   kind: Service
   metadata:
       labels:
           task: monitoring
           #For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
           #If you are NOT using this as an addon, you should comment out this line.
           kubernetes.io/cluster-service: 'true'
           kubernetes.io/name: Heapster
       name: heapster
       namespace: kube-system
   spec:
       ports:
       - port: 80
           targetPort: 8082
       selector:
           k8s-app: heapster

   ##说明

   （1）
   --source 为heapster指定获取集群信息的数据源。参考：https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md
   --sink 为heaster指定后端存储，这里我们使用InfluxDB，其他的，请参考：https://github.com/kubernetes/heapster/blob/master/docs/sink-owners.md
   （2）heapster-rbac.yaml 文件作用
   如没有heapster-rbac.yaml 将导致权限的问题，heaster默认使用一个令×××(Token)与ApiServer进行认证，通过查看heapster.yml发现 serviceAccountName: heapster ，现在明白了吧，就是heaster没有权限，那么如何授权呢-----给heaster绑定一个有权限的角色就行了，即heapster-rbac.yaml配置的那样！

6. 通过dashboard查看集群概况
   
   

7. 通过Grafana查看集群详情(cpu、memory、filesystem、network)
   

8. 文章推荐

k8s 入门教程和实战