云原生架构之Prometheus Operator - 服务自动发现-低调大师

云原生架构之Prometheus Operator - 服务自动发现

2020-12-19 579

一、基于 kubernetes_sd_configs 的动态服务自发现

1.1 定义需要基于 kubernetes_sd_configs 自发现的 Exporter应用，添加 prometheus Annotation labels

apiVersion: v1
kind: Service
metadata:
  name: kafka-message-handler-svc
  annotations:
    prometheus.io/scrape: "true"  # Annotation，用于自发现
    prometheus.io/port: "9080"
    prometheus.io/path: "/prometheus"
spec:
  type: NodePort
  selector:
      app: kafka-message-handler
  ports:
  - name: kafka-message-handler-svc-port
    protocol: TCP
    port: 9080
    targetPort: 9080
#    nodePort: 30287

1.2 配置基于 kubernetes_sd_configs 自发现的 Scrape 数据 pull 配置

新增prometheus在Kubernetes下的自动服务发现 prometheus-additional.yaml

- job_name: 'kubernetes-service'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
- job_name: 'kubernetes-pod'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name
- job_name: 'external-metrics'
  static_configs:
    - targets: ['10.157.227.195:9080']
  metrics_path: '/prometheus'

创建新的 secret additional-configs 从文件 prometheus-additional.yaml

# kubectl delete secret generic additional-configs -n monitoring
kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

更新 Prometheus 部署文件, 新增 additionalScrapeConfigs:> name: additional-configs; key: prometheus-additional.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  retention: 30d
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: prometheus-data-db
        resources:
          requests:
            storage: 30Gi
  baseImage: quay.mirrors.ustc.edu.cn/prometheus/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 1
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0

1.3 测试自发现及相应metrics数据抓取

更新 Prometheus 安装文件，必要的时候可以重启pod

kubectl apply -f prometheus-prometheus.yaml

# 强制重启 kubectl replace --force -f prometheus-prometheus.yaml  or kubectl delete pod -n monitoring prometheus-k8s-0

重启后，在 prometheus 控制台的 status > Targets 及 Service Discovery 没有发现Job kubernetes-service 及 kubernetes-pod，external-metrics Job 出现了，实际上在 Status > Configuration 下面发现 additional-configs 的三个配置是存在的。

如图：

1.4 自发现不生效问题的排查分析

排查分析prometheus日志，发现clusterRole权限不够，日志如下：

level=error ts=2020-12-16T08:58:22.212Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:22.221Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:23.193Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:265: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:23.193Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:264: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:23.215Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:23.223Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:24.195Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:265: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:24.196Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:264: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:24.217Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:24.225Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:25.198Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:265: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:25.199Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:264: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:25.220Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2020-12-16T08:58:25.228Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:283: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

level=info ts=2020-12-16T09:00:03.638Z caller=compact.go:495 component=tsdb msg="write block" mint=1608098400000 maxt=1608105600000 ulid=01ESNCE6T4BJKR81BX12QZFS6S duration=1.393342928s
level=info ts=2020-12-16T09:00:03.884Z caller=head.go:586 component=tsdb msg="head GC completed" duration=129.317903ms
level=info ts=2020-12-16T09:00:05.687Z caller=head.go:656 component=tsdb msg="WAL checkpoint complete" first=26 last=29 duration=1.80239668s
level=info ts=2020-12-16T09:00:07.255Z caller=compact.go:440 component=tsdb msg="compact blocks" count=3 mint=1608076800000 maxt=1608098400000 ulid=01ESNCEA9B4H06MPDCBDWJQJS5 sources="[01ESMQV1240989FW9H7XRD4JG2 01ESMYPRA4Q94FDJM4SWY5E07K 01ESN6CZGGPRFGWHJC49BYKB4F]" duration=1.451881021s

二、使用 CRD ServiceMonitor 来生成 kubernetes_sd_configs 的配置

示例如下

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: cloudnativelabcom
    release: prometheus-operator
  name: cloudnativelabcom
  namespace: monitoring
spec:
  endpoints:
  - interval: 15s
    port: http-metrics
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: cloudnativelabcomapp

实际上，很多的服务是使用 ServiceMonitor 暴露给 Prometheus 抓取的，如下

# kubectl get servicemonitor -A
NAMESPACE    NAME                      AGE
monitoring   alertmanager              27h
monitoring   coredns                   27h
monitoring   grafana                   27h
monitoring   kube-apiserver            27h
monitoring   kube-controller-manager   27h
monitoring   kube-scheduler            27h
monitoring   kube-state-metrics        27h
monitoring   kubelet                   27h
monitoring   nginx                     4h34m
monitoring   node-exporter             27h
monitoring   prometheus                27h
monitoring   prometheus-operator       27h
monitoring   prometheus-pushgateway    27h

- job_name: monitoring/cloudnativelabcom/0
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: cloudnativelabcomapp
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace

三、手工更新

查看当前prometheus 配置（同Prometheus控制台，Status > Configuration）

kubectl get secret -n monitoring prometheus-k8s -o json | jq  -r '.data."prometheus.yaml.gz"' | base64 -d | gzip -d

计划通过从 secret base64解码，解压，导出prometheus.yaml, 让后修改后在压缩，base64编码，edit secret 更新保存，相关实施过程脚本示例如下：

kubectl get secret -n monitoring prometheus-k8s -o json | jq  -r '.data."prometheus.yaml.gz"' | base64 -d | gzip -d > prometheus.yaml
## 修改 vim prometheus.yaml 文件，比如修改既有的配置项，默认拉取时间，比如修改既有的 kubernetes_sd_configs 配置项
gzip prometheus.yaml
base64 prometheus.yaml.gz -w 0 ## -w 0 表示不换行，不成块
kubectl edit secret -n monitoring prometheus-k8s  ## copy 之前 base64 prometheus.yaml.gz -w 0 编码后的值，替换  prometheus.yaml.gz: 的值
## 保存后可再次查看 kubectl get secret -n monitoring prometheus-k8s -o json | jq  -r '.data."prometheus.yaml.gz"' | base64 -d | gzip -d

# 必要时重启生效 kubectl replace --force -f prometheus-prometheus.yaml  or kubectl delete pod -n monitoring prometheus-k8s-0

微信关注我们

原文链接：https://my.oschina.net/neverforget/blog/4816486

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

编程中包、类、变量、方法的命名问题的思考与点滴积累

编程中对于包、类、变量、方法的命名的思考日常开发中对于命名的问题一直是一个头疼且头等重要的问题。好的命名可以很大程度上方便别人的理解与维护，但不合适、粗糙的命名就会将人带入歧途，我自己在工作中也曾深受其害，良久～以下是我自己在日常工作、学习的总结与自己的心得体会，没有百分之百的对与错，只是在现阶段的我看来是最为合适的一种方式。欢迎讨论，但切勿较真。为什么命名会如此痛苦归根结底，我认为是英语文化与汉语文化在表达上的天然差异所造成的。作为使用汉语为母语的Coder，在日常的需求沟通、设计中更多的使用汉语进行表达，但在coding时使用的又是英文，而使用英文去表达汉语的意思时，总是会有一定程度上的失真，据一个典型的例子：四大名著中的《红楼梦》的英译本命名为《A Dream of Red Mansions》或者《The Story of the Stone》，但是这两个翻译都失去了精髓。再者我们对于英文的学习还是停留在纸面上，对英语文化的理解并不多，所以就造成了在使用英文中的一些“近义词”的时候，无法很好的区分它们的细节含义，从而无法更加细致的区分使用哪一个更为合适。在需要梳理逻辑...

2020-12-19

818

在讲述进程之前，先来了解一下二进制可执行文件和目标文件。我们知道程序运行起来后就有了进程，所以了解程序的结构对于认识操作系统大有好处。先抛三个问题：编译器编译代码后生成的文件是目标文件，目标文件里面是什么？可执行文件里面又是什么？可执行文件与目标文件的区别在哪呢？这篇文件会使用如下一个简单的 C 文件进行贯穿讲解： #include <stdio.h> int main(){ printf("hello"); return 0; } 1.编译过程代码要想成为可执行文件就需要经过编译。编译过程分为预处理、编译、汇编、链接。预处理：主要处理源码中的预处理指令，在 C/C++ 中主要指的是“#include”、“#define” 等，它的处理方法是将这些指令所在的位置进行内容替换，比如 “#include” 就会把整个头文件给引进来。编译：把上一步预处理过的文件进行词法分析、语法分析、语义分析&优化一系列步骤后生成汇编代码。这是整个编译过程中最难最复杂的部分。在现在版本的 GCC 中，预处理和编译这两个过程已经合并为一步。汇编：把汇编代码转变为机器可以执行的指...

2020-12-19

519

资源下载

更多资源

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。