安装k8s Master高可用集群-低调大师

安装k8s Master高可用集群

2018-12-15 672

安装k8s Master高可用集群

主机	          角色	            组件

172.18.6.101	K8S Master	Kubelet,kubectl,cni,etcd

172.18.6.102	K8S Master	Kubelet,kubectl,cni,etcd

172.18.6.103	K8S Master	Kubelet,kubectl,cni,etcd

172.18.6.104	K8S Worker	Kubelet,cni

172.18.6.105	K8S Worker	Kubelet,cni

172.18.6.106	K8S Worker	Kubelet,cni

etcd安装

保证k8smaster高可用，不建议使用container的方式启动etcd集群，因为container可能会出现随时死掉的情况，etcd每个节点的启动service又是有状态的。因此此处将以二进制方式进行部署，建议在正式环境中最少部署3个节点的etcd集群，etcd具体安装步骤参考本地服务方式搭建etcd集群

必要组件以及证书安装

ca证书

参考kubernetes中证书生成创建CA证书，并将ca-key.pem与ca.pem放置到k8s集群中所有节点下的/etc/kubernetes/ssl下

woker证书制作

参考kubernetes中证书生成从节点证书生成段落，进行worker节点证书生成。对应ip的证书放置到对应worker节点的/etc/kubernetes/ssl下

kubelet.conf配置安装

创建/etc/kubernetes/kubelet.conf内容如下：

apiVersion: v1

kind: Config

clusters:

- name: local

  cluster:

    server: https://[负载均衡IP]:[apiserver端口]

    certificate-authority: /etc/kubernetes/ssl/ca.pem

users:

- name: kubelet

  user:

    client-certificate: /etc/kubernetes/ssl/worker.pem

    client-key: /etc/kubernetes/ssl/worker-key.pem

contexts:

- context:

    cluster: local

    user: kubelet

  name: kubelet-context

current-context: kubelet-context

cni插件安装

从containernetworking的cni项目中下载cni的必须二进制文件，需要放置到k8s集群中所有节点下的/opt/cni/bin下。

后续将提供rpm包进行一键安装。

kubelet服务部署

注意：后续将提供rpm包进行一键安装。

将对应版本的kubelet二进制文件放置到k8s集群中所有节点下的/usr/bin下

创建/etc/systemd/system/kubelet.service内容如下：

# /etc/systemd/system/kubelet.service

[Unit]

Description=kubelet: The Kubernetes Node Agent

Documentation=http://kubernetes.io/docs/


[Service]

Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"

Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"

Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

Environment="KUBELET_DNS_ARGS=--cluster-dns=10.100.0.10 --cluster-domain=cluster.local"

Environment="KUBELET_EXTRA_ARGS=--pod-infra-container-image=registry.aliyuncs.com/shenshouer/pause-amd64:3.0"

ExecStart=

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS

Restart=always

StartLimitInterval=0

RestartSec=10


[Install]

WantedBy=multi-user.target

创建如下目录：

/etc/kubernetes/

|-- kubelet.conf

|-- manifests

`-- ssl

   |-- ca-key.pem

   |-- ca.pem

   |-- worker.csr

   |-- worker-key.pem

   |-- worker-openssl.cnf

   `-- worker.pem

master组件安装

配置负载均衡

配置LVS使用VIP172.18.6.254指向后端172.18.6.101、172.18.6.102、172.18.6.103，如需简单，则可使用nginx进行TCP4层的负载。

证书生成

openssl.cnf内容如下：

[req]

req_extensions = v3_req

distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = nonRepudiation, digitalSignature, keyEncipherment

subjectAltName = @alt_names

[alt_names]

DNS.1 = kubernetes

DNS.2 = kubernetes.default

DNS.3 = kubernetes.default.svc

DNS.4 = kubernetes.default.svc.cluster.local

DNS.5 = test.example.com.cn

IP.1 = 10.96.0.1

IP.2 = 172.18.6.101

IP.3 = 172.18.6.102

IP.3 = 172.18.6.103

IP.4 = 172.18.6.254

# 三个master的IP

IP.2 = 172.18.6.101

IP.3 = 172.18.6.102

IP.3 = 172.18.6.103

# LVS负载均衡的VIP

IP.4 = 172.18.6.254

# 可能会用到的负载均衡domain

DNS.5 = test.example.com.cn

证书生成具体步骤请参考kubernetes中证书生成 Master证书生成部分与Worker证书生成部分，生成后的证书需要放置到三台Master节点对应路径上

其他组件安装

Master节点上/etc/kubernetes/manifests下放置如下三个文件

kube-apiserver.manifest:

# /etc/kubernetes/manifests/kube-apiserver.manifest

{

  "kind": "Pod",

  "apiVersion": "v1",

  "metadata": {

    "name": "kube-apiserver",

    "namespace": "kube-system",

    "creationTimestamp": null,

    "labels": {

      "component": "kube-apiserver",

      "tier": "control-plane"

    }

  },

  "spec": {

    "volumes": [

      {

        "name": "k8s",

        "hostPath": {

          "path": "/etc/kubernetes"

        }

      },

      {

        "name": "certs",

        "hostPath": {

          "path": "/etc/ssl/certs"

        }

      }

    ],

    "containers": [

      {

        "name": "kube-apiserver",

        "image": "registry.aliyuncs.com.cn/shenshouer/kube-apiserver:v1.5.2",

        "command": [

          "kube-apiserver",

          "--insecure-bind-address=127.0.0.1",

          "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",

          "--service-cluster-ip-range=10.96.0.0/12",

          "--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem",

          "--client-ca-file=/etc/kubernetes/ssl/ca.pem",

          "--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem",

          "--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",

          "--secure-port=6443",

          "--allow-privileged",

          "--advertise-address=[当前Master节点IP]",

          "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",

          "--anonymous-auth=false",

          "--etcd-servers=http://127.0.0.1:2379"

        ],

        "resources": {

          "requests": {

            "cpu": "250m"

          }

        },

        "volumeMounts": [

          {

            "name": "k8s",

            "readOnly": true,

            "mountPath": "/etc/kubernetes/"

          },

          {

            "name": "certs",

            "mountPath": "/etc/ssl/certs"

          }

        ],

        "livenessProbe": {

          "httpGet": {

            "path": "/healthz",

            "port": 8080,

            "host": "127.0.0.1"

          },

          "initialDelaySeconds": 15,

          "timeoutSeconds": 15,

          "failureThreshold": 8

        }

      }

    ],

    "hostNetwork": true

  },

  "status": {}

}

kube-controller-manager.manifest


{

 "kind": "Pod",

 "apiVersion": "v1",

 "metadata": {

   "name": "kube-controller-manager",

   "namespace": "kube-system",

   "creationTimestamp": null,

   "labels": {

     "component": "kube-controller-manager",

     "tier": "control-plane"

   }

 },

 "spec": {

   "volumes": [

     {

       "name": "k8s",

       "hostPath": {

         "path": "/etc/kubernetes"

       }

     },

     {

       "name": "certs",

       "hostPath": {

         "path": "/etc/ssl/certs"

       }

     }

   ],

   "containers": [

     {

       "name": "kube-controller-manager",

       "image": "registry.aliyuncs.com/shenshouer/kube-controller-manager:v1.5.2",

       "command": [

         "kube-controller-manager",

         "--address=127.0.0.1",

         "--leader-elect",

         "--master=127.0.0.1:8080",

         "--cluster-name=kubernetes",

         "--root-ca-file=/etc/kubernetes/ssl/ca.pem",

         "--service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",

         "--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem",

         "--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem",

         "--insecure-experimental-approve-all-kubelet-csrs-for-group=system:kubelet-bootstrap",

         "--allocate-node-cidrs=true",

         "--cluster-cidr=10.244.0.0/16"

       ],

       "resources": {

         "requests": {

           "cpu": "200m"

         }

       },

       "volumeMounts": [

         {

           "name": "k8s",

           "readOnly": true,

           "mountPath": "/etc/kubernetes/"

         },

         {

           "name": "certs",

           "mountPath": "/etc/ssl/certs"

         }

       ],

       "livenessProbe": {

         "httpGet": {

           "path": "/healthz",

           "port": 10252,

           "host": "127.0.0.1"

         },

         "initialDelaySeconds": 15,

         "timeoutSeconds": 15,

         "failureThreshold": 8

       }

     }

   ],

   "hostNetwork": true

 },

 "status": {}

}



kube-scheduler.manifest


{

 "kind": "Pod",

 "apiVersion": "v1",

 "metadata": {

   "name": "kube-scheduler",

   "namespace": "kube-system",

   "creationTimestamp": null,

   "labels": {

     "component": "kube-scheduler",

     "tier": "control-plane"

   }

 },

 "spec": {

   "containers": [

     {

       "name": "kube-scheduler",

       "image": "registry.aliyuncs.com/shenshouer/kube-scheduler:v1.5.2",

       "command": [

         "kube-scheduler",

         "--address=127.0.0.1",

         "--leader-elect",

         "--master=127.0.0.1:8080"

       ],

       "resources": {

         "requests": {

           "cpu": "100m"

         }

       },

       "livenessProbe": {

         "httpGet": {

           "path": "/healthz",

           "port": 10251,

           "host": "127.0.0.1"

         },

         "initialDelaySeconds": 15,

         "timeoutSeconds": 15,

         "failureThreshold": 8

       }

     }

   ],

   "hostNetwork": true

 },

 "status": {}

其他组件安装

kube-proxy安装

在任意master上执行kubectl create -f kube-proxy-ds.yaml，其中kube-proxy-ds.yaml内容如下：

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  labels:

    component: kube-proxy

    k8s-app: kube-proxy

    kubernetes.io/cluster-service: "true"

    name: kube-proxy

    tier: node

  name: kube-proxy

  namespace: kube-system

spec:

  selector:

    matchLabels:

      component: kube-proxy

      k8s-app: kube-proxy

      kubernetes.io/cluster-service: "true"

      name: kube-proxy

      tier: node

  template:

    metadata:

      labels:

        component: kube-proxy

        k8s-app: kube-proxy

        kubernetes.io/cluster-service: "true"

        name: kube-proxy

        tier: node

    spec:

      containers:

      - command:

        - kube-proxy

        - --kubeconfig=/run/kubeconfig

        - --cluster-cidr=10.244.0.0/16

        image: registry.aliyuncs.com/shenshouer/kube-proxy:v1.5.2

        imagePullPolicy: IfNotPresent

        name: kube-proxy

        resources: {}

        securityContext:

          privileged: true

        terminationMessagePath: /dev/termination-log

        volumeMounts:

        - mountPath: /var/run/dbus

          name: dbus

        - mountPath: /run/kubeconfig

          name: kubeconfig

        - mountPath: /etc/kubernetes/ssl

          name: ssl

      dnsPolicy: ClusterFirst

      hostNetwork: true

      restartPolicy: Always

      securityContext: {}

      terminationGracePeriodSeconds: 30

      volumes:

      - hostPath:

          path: /etc/kubernetes/kubelet.conf

        name: kubeconfig

      - hostPath:

          path: /var/run/dbus

        name: dbus

      - hostPath:

          path: /etc/kubernetes/ssl

        name: ssl

网络组件安装

在任意master上执行kubectl apply -f kube-flannel.yaml，其中kube-flannel.yaml内容如下，注意，如果是在vagrant启动的虚拟机中运行，请修改flannled启动参数将--iface指向具体通讯网卡

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: flannel

  namespace: kube-system

---

kind: ConfigMap

apiVersion: v1

metadata:

  namespace: kube-system

  name: kube-flannel-cfg

  labels:

    tier: node

    app: flannel

data:

  cni-conf.json: |

    {

      "name": "cbr0",

      "type": "flannel",

      "delegate": {

        "ipMasq": true,

        "bridge": "cbr0",

        "hairpinMode": true,

        "forceAddress": true,

        "isDefaultGateway": true

      }

    }

  net-conf.json: |

    {

      "Network": "10.244.0.0/16",

      "Backend": {

        "Type": "vxlan"

      }

    }

---

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  namespace: kube-system

  name: kube-flannel-ds

  labels:

    tier: node

    app: flannel

spec:

  template:

    metadata:

      labels:

        tier: node

        app: flannel

    spec:

      hostNetwork: true

      nodeSelector:

        beta.kubernetes.io/arch: amd64

      serviceAccountName: flannel

      containers:

      - name: kube-flannel

        image: registry.aliyuncs.com/shenshouer/flannel:v0.7.0

        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth0" ]

        securityContext:

          privileged: true

        env:

        - name: POD_NAME

          valueFrom:

            fieldRef:

              fieldPath: metadata.name

        - name: POD_NAMESPACE

          valueFrom:

            fieldRef:

              fieldPath: metadata.namespace

        volumeMounts:

        - name: run

          mountPath: /run

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

      - name: install-cni

        image: registry.aliyuncs.com/shenshouer/flannel:v0.7.0

        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]

        volumeMounts:

        - name: cni

          mountPath: /etc/cni/net.d

        - name: flannel-cfg

          mountPath: /etc/kube-flannel/

      volumes:

        - name: run

          hostPath:

            path: /run

        - name: cni

          hostPath:

            path: /etc/cni/net.d

        - name: flannel-cfg

          configMap:

            name: kube-flannel-cfg

DNS部署

在任意master上执行kubectl create -f skydns.yaml，其中skydns.yaml内容如下

apiVersion: v1

kind: Service

metadata:

  name: kube-dns

  namespace: kube-system

  labels:

    k8s-app: kube-dns

    kubernetes.io/cluster-service: "true"

    kubernetes.io/name: "KubeDNS"

spec:

  selector:

    k8s-app: kube-dns

  clusterIP: 10.100.0.10

  ports:

  - name: dns

    port: 53

    protocol: UDP

  - name: dns-tcp

    port: 53

    protocol: TCP


---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: kube-dns

  namespace: kube-system

  labels:

    k8s-app: kube-dns

    kubernetes.io/cluster-service: "true"

spec:

  # replicas: not specified here:

  # 1. In order to make Addon Manager do not reconcile this replicas parameter.

  # 2. Default is 1.

  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.

  strategy:

    rollingUpdate:

      maxSurge: 10%

      maxUnavailable: 0

  selector:

    matchLabels:

      k8s-app: kube-dns

  template:

    metadata:

      labels:

        k8s-app: kube-dns

      annotations:

        scheduler.alpha.kubernetes.io/critical-pod: ''

        scheduler.alpha.kubernetes.io/tolerations: '[{"key":"CriticalAddonsOnly", "operator":"Exists"}]'

    spec:

      containers:

      - name: kubedns

        image: registry.aliyuncs.com/shenshouer/kubedns-amd64:1.9

        resources:

          # TODO: Set memory limits when we've profiled the container for large

          # clusters, then set request = limit to keep this container in

          # guaranteed class. Currently, this container falls into the

          # "burstable" category so the kubelet doesn't backoff from restarting it.

          limits:

            memory: 170Mi

          requests:

            cpu: 100m

            memory: 70Mi

        livenessProbe:

          httpGet:

            path: /healthz-kubedns

            port: 8080

            scheme: HTTP

          initialDelaySeconds: 60

          timeoutSeconds: 5

          successThreshold: 1

          failureThreshold: 5

        readinessProbe:

          httpGet:

            path: /readiness

            port: 8081

            scheme: HTTP

          # we poll on pod startup for the Kubernetes master service and

          # only setup the /readiness HTTP server once that's available.

          initialDelaySeconds: 3

          timeoutSeconds: 5

        args:

        - --domain=cluster.local.

        - --dns-port=10053

        - --config-map=kube-dns

        # This should be set to v=2 only after the new image (cut from 1.5) has

        # been released, otherwise we will flood the logs.

        - --v=0

        - --federations=myfederation=federation.test

        env:

        - name: PROMETHEUS_PORT

          value: "10055"

        ports:

        - containerPort: 10053

          name: dns-local

          protocol: UDP

        - containerPort: 10053

          name: dns-tcp-local

          protocol: TCP

        - containerPort: 10055

          name: metrics

          protocol: TCP

      - name: dnsmasq

        image: registry.aliyuncs.com/shenshouer/kube-dnsmasq-amd64:1.4

        livenessProbe:

          httpGet:

            path: /healthz-dnsmasq

            port: 8080

            scheme: HTTP

          initialDelaySeconds: 60

          timeoutSeconds: 5

          successThreshold: 1

          failureThreshold: 5

        args:

        - --cache-size=1000

        - --no-resolv

        - --server=127.0.0.1#10053

        - --log-facility=-

        ports:

        - containerPort: 53

          name: dns

          protocol: UDP

        - containerPort: 53

          name: dns-tcp

          protocol: TCP

        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details

        resources:

          requests:

            cpu: 150m

            memory: 10Mi

      - name: dnsmasq-metrics

        image: registry.aliyuncs.com/shenshouer/dnsmasq-metrics-amd64:1.0

        livenessProbe:

          httpGet:

            path: /metrics

            port: 10054

            scheme: HTTP

          initialDelaySeconds: 60

          timeoutSeconds: 5

          successThreshold: 1

          failureThreshold: 5

        args:

        - --v=2

        - --logtostderr

        ports:

        - containerPort: 10054

          name: metrics

          protocol: TCP

        resources:

          requests:

            memory: 10Mi

      - name: healthz

        image: registry.aliyuncs.com/shenshouer/exechealthz-amd64:1.2

        resources:

          limits:

            memory: 50Mi

          requests:

            cpu: 10m

            # Note that this container shouldn't really need 50Mi of memory. The

            # limits are set higher than expected pending investigation on #29688.

            # The extra memory was stolen from the kubedns container to keep the

            # net memory requested by the pod constant.

            memory: 50Mi

        args:

        - --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null

        - --url=/healthz-dnsmasq

        - --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null

        - --url=/healthz-kubedns

        - --port=8080

        - --quiet

        ports:

        - containerPort: 8080

          protocol: TCP

      dnsPolicy: Default  # Don't use cluster DNS.

Node节点安装

Docker安装

新建/etc/kubernetes/目录


|-- kubelet.conf

|-- manifests

`-- ssl

  |-- ca-key.pem

  |-- ca.pem

  |-- ca.srl

  |-- worker.csr

  |-- worker-key.pem

  |-- worker-openssl.cnf

  `-- worker.pem

新建/etc/kubernetes/kubelet.conf配置，参考kubelet.conf配置

新建/etc/kubernetes/ssl，证书制作参考worker证书制作

新建/etc/kubernetes/manifests

新建/opt/cni/bin,安装CNI参考cni安装步骤

安装kubelet,参考kubelet安装

systemctl enable kubelet && systemctl restart kubelet && journalctl -fu kubelet

本文转自CSDN-安装k8s Master高可用集群

微信关注我们

原文链接：https://yq.aliyun.com/articles/679672

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

自己已备案的服务器做代理服务器将网站的请求转发到ECS上ECS是否需要备案

我现在有一台自己购买的服务器，有一个已在非万网备案的域名，该域名与我自己购买的服务器绑定，我现在如果将自己的服务器作为代理服务器将网站请求转发到租用的ECS上（通过ip地址），我这个ECS是否需要备案？或者说是否会被工信部或阿里屏蔽？目前规则要求如果您的域名直接访问阿里云服务器IP便需要备案，如您只是使用IP访问，无需进行备案，请您了解~

2018-12-16

648

k8s是什么:Docker分布式系统解决方案单机版k8s环境搭建：centos7 systemctl disabled firewalld systemctl stop firewalld 安装etcd（高可用键值数据库）和k8s yum update #多次尝试，确认网络连接 yum install -y etcd kubernetes 启动服务（进程） systemctl start etcd systemctl start docker #容器创建管理 systemctl start kube-apiserver #所有资源增、删、改、查等操作入口 systemctl start kube-controller-manager #自动化控制中心 systemctl start kube-scheduler #pod调度 systemctl start kubelet #node节点进程，负责pod对应容器创建、起停 sustemctl start kube-proxy #service通信及负载均衡机制组件常用操作命令 kubectl get nodes #查看集群中nod...

2018-12-16

674

资源下载

更多资源

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。