您现在的位置是：首页 > 文章详情

kubernetes部署Ingress访问代理与负载均衡器

日期：2018-12-17点击：554收藏

Kubernetes中的pod都有独立的内部IP（外部不可访问），通过Service可以对多个pod进行负载均衡和故障转移，Service可以具有ClusterIP、NodeIP或LoadBanlancer模式。目前，ClusterIP只能内部访问，需通过kubectl proxy代理出来，NodeIP是跟Node绑定的、迁移性差，LoadBanlancer的每个服务都有独立的IP地址，管理、使用不便。有没有一个固定的独立IP、自动节点漂移的解决方案呢？以前这样的功能基本上都用Nginx来实现，现在Kubernetes有一个做好了的服务，也是基于Nginx的，就是Ingress。

==============================================================================

以下内容来自：http://blog.51cto.com/newfly/2060587

如何访问K8S中的服务：

1、Ingress介绍

Kubernetes 暴露服务的方式目前只有三种：LoadBlancer Service、NodePort Service、Ingress；前两种估计都应该很熟悉，下面详细的了解下这个 Ingress

Ingress由两部分组成：Ingress Controller 和 Ingress 服务。

Ingress Contronler 通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取它，按照自定义的规则，规则就是写明了哪个域名对应哪个service，生成一段 Nginx 配置，再写到 Nginx-ingress-control的 Pod 里，这个 Ingress Contronler 的pod里面运行着一个nginx服务，控制器会把生成的nginx配置写入/etc/nginx.conf文件中，然后 reload 一下使用配置生效。以此来达到域名分配置及动态更新的问题。

看个简单的图方便理解：

ingress控制器有两种：nginx和haproxy 这里是以nginx为讲解。

2、部署一个Nginx Ingress

ingress的部署文件在github Ingress 仓库找到. 针对官方配置我们单独添加了 nodeselector 指定，绑定LB地址以方便DNS 做解析。

主要用到的文件：

$ ls

default-backend.yaml jenkins-ingress.yml nginx-ingress-controller-rbac.yml nginx-ingress-controller.yaml

- - -

default-backend.yaml：这是官方要求必须要给的默认后端，提供404页面的。它还提供了一个http检测功能，检测nginx-ingress-controll健康状态的，通过每隔一定时间访问nginx-ingress-controll的/healthz页面，如是没有响应就

返回404之类的错误码。

nginx-ingress-controller-rbac.yml:这ingress的RBAC授权文件

nginx-ingress-controller.yaml：这是控制器的部署文件。

jenkins-ingress.yml：这是Ingress服务文件，这个可以是任意web程序，里面配置域名与service的对应关系，Ingress称之为规则。

第一个是要部署RBAC文件：

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

cat nginx-ingress-controller-rbac.yml

#apiVersion: v1

#kind: Namespace

#metadata: #这里是创建一个namespace，因为此namespace早有了就不用再创建了

# name: kube-system

---

apiVersion: v1

kind: ServiceAccount

metadata:

name: nginx-ingress-serviceaccount #创建一个serveerAcount

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRole

metadata:

name: nginx-ingress-clusterrole #这个ServiceAcount所绑定的集群角色

rules:

- apiGroups:

- ""

resources: #此集群角色的权限，它能操作的API资源

- configmaps

- endpoints

- nodes

- pods

- secrets

verbs:

- list

- watch

- apiGroups:

- ""

resources:

- nodes

verbs:

- get

- apiGroups:

- ""

resources:

- services

verbs:

- get

- list

- watch

- apiGroups:

- "extensions"

resources:

- ingresses

verbs:

- get

- list

- watch

- apiGroups:

- ""

resources:

- events

verbs:

- create

- patch

- apiGroups:

- "extensions"

resources:

- ingresses/status

verbs:

- update

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: Role

metadata:

name: nginx-ingress-role #这是一个角色，而非集群角色

namespace: kube-system

rules: #角色的权限

- apiGroups:

- ""

resources:

- configmaps

- pods

- secrets

- namespaces

verbs:

- get

- apiGroups:

- ""

resources:

- configmaps

resourceNames:

# Defaults to "<election-id>-<ingress-class>"

# Here: "<ingress-controller-leader>-<nginx>"

# This has to be adapted if you change either parameter

# when launching the nginx-ingress-controller.

- "ingress-controller-leader-nginx"

verbs:

- get

- update

- apiGroups:

- ""

resources:

- configmaps

verbs:

- create

- apiGroups:

- ""

resources:

- endpoints

verbs:

- get

- create

- update

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: RoleBinding #角色绑定

metadata:

name: nginx-ingress-role-nisa-binding

namespace: kube-system

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

name: nginx-ingress-role

subjects:

- kind: ServiceAccount

name: nginx-ingress-serviceaccount #绑定在这个用户

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding #集群绑定

metadata:

name: nginx-ingress-clusterrole-nisa-binding

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: nginx-ingress-clusterrole

subjects:

- kind: ServiceAccount

name: nginx-ingress-serviceaccount #集群绑定到这个serviceacount

namespace: kube-system #集群角色是可以跨namespace，但是这里只指明给这个namespce来使用

创建：

$ kubectl create -f nginx-ingress-controller-rbac.yml

serviceaccount "nginx-ingress-serviceaccount" created

clusterrole "nginx-ingress-clusterrole" created

role "nginx-ingress-role" created

rolebinding "nginx-ingress-role-nisa-binding" created

clusterrolebinding "nginx-ingress-clusterrole-nisa-binding" created

RBAC创建完后，就创建default backend服务：

$ cat default-backend.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: default-http-backend

labels:

k8s-app: default-http-backend

namespace: kube-system

spec:

replicas: 1

template:

metadata:

labels:

k8s-app: default-http-backend

spec:

terminationGracePeriodSeconds: 60

containers:

- name: default-http-backend

# Any image is permissable as long as:

# 1. It serves a 404 page at /

# 2. It serves 200 on a /healthz endpoint

image: gcr.io/google_containers/defaultbackend:1.0

livenessProbe:

httpGet:

path: /healthz #这个URI是 nginx-ingress-controller中nginx里配置好的localtion

port: 8080

scheme: HTTP

initialDelaySeconds: 30 #30s检测一次/healthz

timeoutSeconds: 5

ports:

- containerPort: 8080

resources:

limits:

cpu: 10m

memory: 20Mi

requests:

cpu: 10m

memory: 20Mi

nodeSelector: #指定调度到些Node, 以便后面DNS解析

kubernetes.io/hostname: 10.3.1.17

---

apiVersion: v1

kind: Service #为default backend 创建一个service

metadata:

name: default-http-backend

namespace: kube-system

labels:

k8s-app: default-http-backend

spec:

ports:

- port: 80

targetPort: 8080

selector:

k8s-app: default-http-backend

创建：

$ kubectl create -f default-backend.yaml

deployment "default-http-backend" created

service "default-http-backend" created

创建之后查看：

root@ubuntu15:/data/ingress# kubectl get rs,pod,svc -n kube-system

NAME DESIRED CURRENT READY AGE

rs/default-http-backend-857b544d94 1 1 1 1m

NAME READY STATUS RESTARTS AGE

po/default-http-backend-857b544d94-bwgjd 1/1 Running 0 1m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

svc/default-http-backend ClusterIP 10.254.208.144 <none> 80/TCP 1m

创建好default backend后就要创建nginx-ingress-controller了：

$ cat nginx-ingress-controller.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-ingress-controller labels: k8s-app: nginx-ingress-controller namespace: kube-system spec: replicas: 1 template: metadata: labels: k8s-app: nginx-ingress-controller spec: # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used # like with kubeadm # hostNetwork: true #注释表示不使用宿主机的80口， terminationGracePeriodSeconds: 60 hostNetwork: true #表示容器使用和宿主机一样的网络 serviceAccountName: nginx-ingress-serviceaccount #引用前面创建的serviceacount containers: - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 #容器使用的镜像 name: nginx-ingress-controller #容器名 readinessProbe: #启动这个服务时要验证/healthz 端口10254会在运行的node上监听。  httpGet: path: /healthz port: 10254 scheme: HTTP livenessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 #每隔10做健康检查  timeoutSeconds: 1 ports: - containerPort: 80 hostPort: 80 #80映射到80 - containerPort: 443 hostPort: 443 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend # - --default-ssl-certificate=$(POD_NAMESPACE)/ingress-secret #这是启用Https时用的 nodeSelector: #指明运行在哪，此IP要和default backend是同一个IP kubernetes.io/hostname: 10.3.1.17 #上面映射到了hostport80，确保此IP80，443没有占用.

这个控制器就是一个deployment ，里面运行一个容器gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 ，有点像nginx容器，现在创建：

1 2	`$ kubectl create -f nginx-ingress-controller.yaml` `deployment` `"nginx-ingress-controller"` `created`

root@ubuntu15:/data/ingress# kubectl get rs,pod,svc -n kube-system

NAME DESIRED CURRENT READY AGE

rs/default-http-backend-857b544d94 1 1 1 12m

rs/nginx-ingress-controller-8576d4545d 1 1 0 27s

NAME READY STATUS RESTARTS AGE

po/default-http-backend-857b544d94-bwgjd 1/1 Running 0 12m

po/nginx-ingress-controller-8576d4545d-9tjnv 0/1 ContainerCreating 0 27s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

svc/default-http-backend ClusterIP 10.254.208.144 <none> 80/TCP 12m

现在ingress controller 控制器已部署好了，那么如何使用了，那就要写一个ingress规则了，此处就以已存在的jenkins服务为例，配置如何使用域名访问这个service:

$ kubectl get svc,ep

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

svc/jenkinsservice NodePort 10.254.70.47 <none> 8080:30002/TCP 3h

NAME ENDPOINTS AGE

ep/jenkinsservice 172.30.10.15:8080,172.30.11.7:8080 3h

现在写个jenkins service的Ingress 规则：

$ cat jenkins-ingress.yml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: jenkins-ingress

namespace: default #服务在哪个空间内就写哪个空间

annotations:

kubernetes.io/ingress.class: "nginx"

spec:

rules:

- host: ingress.jenkins.com #此service的访问域名

http:

paths:

- backend:

serviceName: jenkinsservice

servicePort: 8080

创建它：

$ kubectl create -f jenkins-ingress.yml ingress "jenkins-ingress" created $ kubectl get ingress NAME HOSTS ADDRESS PORTS AGE jenkins-ingress ingress.jenkins.com 80 10s

到这里就已经部署完成了，配置好域名后，就可以用此域名来访问了：

部署完成了，现在看下nginx-ingress-controller 里nginx配置文件发生了哪些变化：

upstream default-jenkinsservice-8080 { least_conn; server 172.30.10.15:8080 max_fails=0 fail_timeout=0; server 172.30.11.7:8080 max_fails=0 fail_timeout=0; } upstream upstream-default-backend { least_conn; server 172.30.11.6:8080 max_fails=0 fail_timeout=0; } server { server_name ingress.jenkins.com; listen [::]:80; location / { ... proxy_pass http://default-jenkinsservice-8080; ... } }