详解K8s 镜像缓存管理kube-fledged-低调大师

详解K8s 镜像缓存管理kube-fledged

2024-04-15 299

本文分享自华为云社区《K8s 镜像缓存管理 kube-fledged 认知》，作者：山河已无恙。

我们知道 k8s 上的容器调度需要在调度的节点行拉取当前容器的镜像，在一些特殊场景中，

需要 快速启动和/或扩展的应用程序。例如，由于数据量激增，执行实时数据处理的应用程序需要快速扩展。
镜像比较庞大，涉及多个版本，节点存储有限，需要动态清理不需要的镜像
无服务器函数通常需要在几分之一秒内立即对传入事件和启动容器做出反应。
在边缘设备上运行的 IoT 应用程序，需要容忍 边缘设备和镜像镜像仓库之间的间歇性网络连接。
如果需要从 专用仓库中拉取镜像，并且无法授予每个人从此 镜像仓库拉取镜像的访问权限，则可以在群集的节点上提供镜像。
如果集群管理员或操作员需要对应用程序进行升级，并希望事先验证是否可以成功拉取新镜像。

kube-fledged 是一个 kubernetes operator，用于直接在 Kubernetes 集群的 worker 节点上创建和管理容器镜像缓存。它允许用户定义镜像列表以及这些镜像应缓存到哪些工作节点上（即拉取）。因此，应用程序 Pod 几乎可以立即启动，因为不需要从镜像仓库中提取镜像。

kube-fledged 提供了 CRUD API 来管理镜像缓存的生命周期，并支持多个可配置的参数，可以根据自己的需要自定义功能。

Kubernetes 具有内置的镜像垃圾回收机制。节点中的 kubelet 会定期检查磁盘使用率是否达到特定阈值（可通过标志进行配置）。一旦达到这个阈值，kubelet 会自动删除节点中所有未使用的镜像。

需要在建议的解决方案中实现自动和定期刷新机制。如果镜像缓存中的镜像被 kubelet 的 gc 删除，下一个刷新周期会将已删除的镜像拉入镜像缓存中。这可确保镜像缓存是最新的。

设计流程

https://github.com/senthilrch/kube-fledged/blob/master/docs/kubefledged-architecture.png

部署 kube-fledged

Helm 方式部署

──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$mkdir  kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$cd kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$export KUBEFLEDGED_NAMESPACE=kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$kubectl create namespace ${KUBEFLEDGED_NAMESPACE}
namespace/kube-fledged created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/
"kubefledged-charts" has been added to your repositories
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kubefledged-charts" chart repository
...Successfully got an update from the "kubescape" chart repository
...Successfully got an update from the "rancher-stable" chart repository
...Successfully got an update from the "skm" chart repository
...Successfully got an update from the "openkruise" chart repository
...Successfully got an update from the "awx-operator" chart repository
...Successfully got an update from the "botkube" chart repository
Update Complete. ⎈Happy Helming!⎈

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$helm install --verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait

实际部署中发现，由于网络问题，chart 无法下载，所以通过 make deploy-using-yaml 使用 yaml 方式部署

Yaml 文件部署

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$git clone https://github.com/senthilrch/kube-fledged.git
正克隆到 'kube-fledged'...
remote: Enumerating objects: 10613, done.
remote: Counting objects: 100% (1501/1501), done.
remote: Compressing objects: 100% (629/629), done.
remote: Total 10613 (delta 845), reused 1357 (delta 766), pack-reused 9112
接收对象中: 100% (10613/10613), 34.58 MiB | 7.33 MiB/s, done.
处理 delta 中: 100% (4431/4431), done.
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$ls
kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$cd kube-fledged/
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$make deploy-using-yaml
kubectl apply -f deploy/kubefledged-namespace.yaml

第一次部署，发现镜像拉不下来

┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get all -n kube-fledged
NAME                                               READY   STATUS                  RESTARTS         AGE
pod/kube-fledged-controller-df69f6565-drrqg        0/1     CrashLoopBackOff        35 (5h59m ago)   21h
pod/kube-fledged-webhook-server-7bcd589bc4-b7kg2   0/1     Init:CrashLoopBackOff   35 (5h58m ago)   21h
pod/kubefledged-controller-55f848cc67-7f4rl        1/1     Running                 0                21h
pod/kubefledged-webhook-server-597dbf4ff5-l8fbh    0/1     Init:CrashLoopBackOff   34 (6h ago)      21h

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   21h
service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   21h

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-fledged-controller       0/1     1            0           21h
deployment.apps/kube-fledged-webhook-server   0/1     1            0           21h
deployment.apps/kubefledged-controller        0/1     1            0           21h
deployment.apps/kubefledged-webhook-server    0/1     1            0           21h

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/kube-fledged-controller-df69f6565        1         1         0       21h
replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         0       21h
replicaset.apps/kubefledged-controller-55f848cc67        1         1         0       21h
replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         0       21h
┌──[root@vms100.liruilongs.github.io]-[~]
└─$

这里我们找一下要拉取的镜像

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat *.yaml | grep image:
      - image: senthilrch/kubefledged-controller:v0.10.0
      - image: senthilrch/kubefledged-webhook-server:v0.10.0
      - image: senthilrch/kubefledged-webhook-server:v0.10.0

单独拉取一些，当前使用 ansible 在所有工作节点批量操作

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible k8s_node -m shell -a "docker pull docker.io/senthilrch/kubefledged-cri-client:v0.10.0" -i host.yaml

其他相关的镜像都拉取一下

操作完成之后容器状态全部正常

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl -n kube-fledged get all
NAME                                               READY   STATUS    RESTARTS   AGE
pod/kube-fledged-controller-df69f6565-wdb4g        1/1     Running   0          13h
pod/kube-fledged-webhook-server-7bcd589bc4-j8xxp   1/1     Running   0          13h
pod/kubefledged-controller-55f848cc67-klxlm        1/1     Running   0          13h
pod/kubefledged-webhook-server-597dbf4ff5-ktbsh    1/1     Running   0          13h

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   36h
service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   36h

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-fledged-controller       1/1     1            1           36h
deployment.apps/kube-fledged-webhook-server   1/1     1            1           36h
deployment.apps/kubefledged-controller        1/1     1            1           36h
deployment.apps/kubefledged-webhook-server    1/1     1            1           36h

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/kube-fledged-controller-df69f6565        1         1         1       36h
replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         1       36h
replicaset.apps/kubefledged-controller-55f848cc67        1         1         1       36h
replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         1       36h

验证是否安装成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$kubectl get pods -n kube-fledged -l app=kubefledged
NAME                                          READY   STATUS    RESTARTS   AGE
kubefledged-controller-55f848cc67-klxlm       1/1     Running   0          16h
kubefledged-webhook-server-597dbf4ff5-ktbsh   1/1     Running   0          16h
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$kubectl get imagecaches -n kube-fledged
No resources found in kube-fledged namespace.

使用 kubefledged

创建镜像缓存对象

根据 Demo 文件，创建镜像缓存对象

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$cd deploy/
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat kubefledged-imagecache.yaml
---
apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
metadata:
  # Name of the image cache. A cluster can have multiple image cache objects
  name: imagecache1
  namespace: kube-fledged
  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  labels:
    app: kubefledged
    kubefledged: imagecache
spec:
  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  cacheSpec:
  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  - images:
    - ghcr.io/jitesoft/nginx:1.23.1
  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  - images:
    - us.gcr.io/k8s-artifacts-prod/cassandra:v7
    - us.gcr.io/k8s-artifacts-prod/etcd:3.5.4-0
    nodeSelector:
      tier: backend
  # Specifies a list of image pull secrets to pull images from private repositories into the cache
  imagePullSecrets:
  - name: myregistrykey

官方的 Demo 中对应的镜像拉取不下来，所以换一下

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$docker pull us.gcr.io/k8s-artifacts-prod/cassandra:v7
Error response from daemon: Get "https://us.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

为了测试选择器标签的使用,我们找一个节点的标签单独做镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl get nodes  --show-labels

同时我们直接从公有仓库拉取镜像，所以不需要 imagePullSecrets 对象

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$vim kubefledged-imagecache.yaml

修改后的 yaml 文件

添加了一个所有节点的 liruilong/my-busybox:latest 镜像缓存
添加了一个 kubernetes.io/hostname: vms105.liruilongs.github.io 对应标签选择器的 liruilong/hikvision-sdk-config-ftp:latest 镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat kubefledged-imagecache.yaml
---
apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
metadata:
  # Name of the image cache. A cluster can have multiple image cache objects
  name: imagecache1
  namespace: kube-fledged
  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  labels:
    app: kubefledged
    kubefledged: imagecache
spec:
  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  cacheSpec:
  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  - images:
    - liruilong/my-busybox:latest
  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  - images:
    - liruilong/hikvision-sdk-config-ftp:latest
    nodeSelector:
      kubernetes.io/hostname: vms105.liruilongs.github.io
  # Specifies a list of image pull secrets to pull images from private repositories into the cache
  #imagePullSecrets:
  #- name: myregistrykey
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

直接创建报错了

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl create -f kubefledged-imagecache.yaml
Error from server (InternalError): error when creating "kubefledged-imagecache.yaml": Internal error occurred: failed calling webhook "validate-image-cache.kubefledged.io": failed to call webhook: Post "https://kubefledged-webhook-server.kube-fledged.svc:3443/validate-image-cache?timeout=1s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefledged.io")
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl get imagecaches -n kube-fledged
No resources found in kube-fledged namespace.
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

解决办法，删除对应的对象，重新创建

我在当前项目的一个 issues 下面找到了解决办法 https://github.com/senthilrch/kube-fledged/issues/76

看起来这是因为 Webhook CA 是硬编码的，但是当 webhook 服务器启动时，会生成一个新的 CA 捆绑包并更新 webhook 配置。当发生另一个部署时，将重新应用原始 CA 捆绑包，并且 Webhook 请求开始失败，直到再次重新启动 Webhook 组件以修补捆绑包init-server

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$make remove-kubefledged-and-operator
# Remove kubefledged
kubectl delete -f deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml
error: resource mapping not found for name: "kube-fledged" namespace: "kube-fledged" from "deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml": no matches for kind "KubeFledged" in version "charts.helm.kubefledged.io/v1alpha2"
ensure CRDs are installed first

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$make deploy-using-yaml
kubectl apply -f deploy/kubefledged-namespace.yaml
namespace/kube-fledged created
kubectl apply -f deploy/kubefledged-crd.yaml
customresourcedefinition.apiextensions.k8s.io/imagecaches.kubefledged.io unchanged
....................
kubectl rollout status deployment kubefledged-webhook-server -n kube-fledged --watch
Waiting for deployment "kubefledged-webhook-server" rollout to finish: 0 of 1 updated replicas are available...
deployment "kubefledged-webhook-server" successfully rolled out
kubectl get pods -n kube-fledged
NAME                                          READY   STATUS    RESTARTS   AGE
kubefledged-controller-55f848cc67-76c4v       1/1     Running   0          112s
kubefledged-webhook-server-597dbf4ff5-56h6z   1/1     Running   0          66s

重新创建缓存对象，创建成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl create -f kubefledged-imagecache.yaml
imagecache.kubefledged.io/imagecache1 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl get imagecaches -n kube-fledged
NAME          AGE
imagecache1   10s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

查看当前被纳管的镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$kubectl get imagecaches imagecache1 -n kube-fledged -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 83,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20169836",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    "liruilong/my-busybox:latest"
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "completionTime": "2024-03-02T01:06:47Z",
        "message": "All requested images pulled succesfully to respective nodes",
        "reason": "ImageCacheRefresh",
        "startTime": "2024-03-02T01:05:33Z",
        "status": "Succeeded"
    }
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$

通过 ansible 来验证

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
192.168.26.102 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.101 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.103 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.105 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.100 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.106 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/hikvision-sdk-config-ftp" -i host.yaml
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.105 | CHANGED | rc=0 >>
liruilong/hikvision-sdk-config-ftp                                          latest            a02cd03b4342   4 months ago    830MB
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

开启自动刷新

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache=
imagecache.kubefledged.io/imagecache1 annotated
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

添加镜像缓存

添加一个新的镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 92,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20175233",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    "liruilong/my-busybox:latest",
                    "liruilong/jdk1.8_191:latest"
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "completionTime": "2024-03-02T01:43:32Z",
        "message": "All requested images pulled succesfully to respective nodes",
        "reason": "ImageCacheUpdate",
        "startTime": "2024-03-02T01:40:34Z",
        "status": "Succeeded"
    }
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

通过 ansible 确认

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
192.168.26.101 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
192.168.26.102 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
192.168.26.100 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
192.168.26.103 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest                                      17dbd4002a8c   5 years ago     170MB
192.168.26.105 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB
192.168.26.106 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

删除镜像缓存

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl edit imagecaches imagecache1 -n kube-fledged
imagecache.kubefledged.io/imagecache1 edited
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 94,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20175766",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    "liruilong/jdk1.8_191:latest"
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "message": "Image cache is being updated. Please view the status after some time",
        "reason": "ImageCacheUpdate",
        "startTime": "2024-03-02T01:48:03Z",
        "status": "Processing"
    }
}

通过 Ansible 确认，可以看到无论是 mastere 上的节点还是 work 的节点，对应的镜像缓存都被清理

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
192.168.26.102 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.101 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

这里需要注意如果清除所有的镜像缓存，那么需要把 images 下的数组写成 "".

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl edit imagecaches imagecache1 -n kube-fledged
imagecache.kubefledged.io/imagecache1 edited
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 98,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20176849",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    ""
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "completionTime": "2024-03-02T01:52:16Z",
        "message": "All cached images succesfully deleted from respective nodes",
        "reason": "ImageCacheUpdate",
        "startTime": "2024-03-02T01:51:47Z",
        "status": "Succeeded"
    }
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

如果通过下面的方式删除，直接注释调对应的标签

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat kubefledged-imagecache.yaml
---
apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
metadata:
  # Name of the image cache. A cluster can have multiple image cache objects
  name: imagecache1
  namespace: kube-fledged
  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  labels:
    app: kubefledged
    kubefledged: imagecache
spec:
  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  cacheSpec:
  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  #- images:
    #- liruilong/my-busybox:latest
  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  - images:
    - liruilong/hikvision-sdk-config-ftp:latest
    nodeSelector:
      kubernetes.io/hostname: vms105.liruilongs.github.io
  # Specifies a list of image pull secrets to pull images from private repositories into the cache
  #imagePullSecrets:
  #- name: myregistrykey
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

那么会报下面的错

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl edit imagecaches imagecache1 -n kube-fledged
error: imagecaches.kubefledged.io "imagecache1" could not be patched: admission webhook "validate-image-cache.kubefledged.io" denied the request: Mismatch in no. of image lists
You can run `kubectl replace -f /tmp/kubectl-edit-4113815075.yaml` to try this update again.

博文部分内容参考

https://github.com/senthilrch/kube-fledged

点击关注，第一时间了解华为云新鲜技术~

微信关注我们

原文链接：https://my.oschina.net/u/4526289/blog/11052468

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

基于香橙派AIpro将开源框架模型转换为昇腾模型

本文分享自华为云社区《如何基于香橙派AIpro将开源框架模型转换为昇腾模型》，作者：昇腾CANN。在前面的介绍中，我们知道了如何基于香橙派AIpro开发AI推理应用，也大致了解到在推理之前，需要把原始网络模型 (可能是 PyTorch 的、TensorFlow，可能是Caffe的等等) 转换成 .om 模型，然后才能调用昇腾的aclmdlExecute 等模型执行接口在香橙派AIpro进行模型推理。这个模型转换的过程就要用到 ATC 工具，目前 ATC 工具直接支持从 Caffe、ONNX、TensorFlow 以及 MindSpore模型的转换，所以如果你的训练框架是 PyTorch，则需要做 torch.onnx.export 操作导出成ONNX模型后才能使用ATC工具。 01 ATC工具简介昇腾张量编译器（Ascend Tensor Compiler，简称ATC）是昇腾模型转换工具，它可以将开源框架的网络模型（例如TensorFlow、ONNX等）转换为昇腾AI处理器支持的模型文件（.om格式），用于后续的模型推理。模型转换过程中，ATC会进行算子调度优化、权重数据重排、内...

2024-04-12

278

本文分享自华为云社区《解锁物联网数据管理的新境界：华为云OBS助力数据转发与存储【零代码零硬件玩转华为云IoT】》，作者： Y-StarryDreamer。 1. 背景介绍物联网（IoT）技术的快速发展正在推动着智能化社会的到来。在这个数字化时代，随着物联网设备数量的不断增加和数据规模的快速增长，如何有效地存储和管理这些海量数据成为了一个重要的挑战。华为云作为领先的云计算服务提供商，积极响应这一挑战，为用户提供了丰富的物联网解决方案，其中包括对象存储服务（OBS）。 OBS作为一种高度可扩展、安全可靠的云端存储服务，为用户提供了灵活、高效的数据存储解决方案。它采用分布式存储架构，能够轻松应对海量数据的存储需求，并提供了多种数据保护和安全控制功能，确保用户的数据安全可靠。在物联网应用中，数据的长期存储和管理是至关重要的。传统的数据存储方式往往需要用户自行搭建和维护服务器，成本高昂且管理复杂。而将数据转发至OBS，则成为了一种更加便捷和经济的解决方案。通过将数据存储在OBS中，用户可以充分利用其高可靠性和低成本的优势，无需担心数据丢失或存储空间不足的问题。同时，OBS还提供了丰富的数据...

2024-04-15

314

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。