Kubernetes中的pod都有独立的内部IP（外部不可访问），通过Service可以对多个pod进行负载均衡和故障转移，Service可以具有ClusterIP、NodeIP或LoadBanlancer模式。目前，ClusterIP只能内部访问，需通过kubectl proxy代理出来，NodeIP是跟Node绑定的、迁移性差，LoadBanlancer的每个服务都有独立的IP地址，管理、使用不便。有没有一个固定的独立IP、自动节点漂 移的解决方案呢？以前这样的功能基本上都用Nginx来实现，现在Kubernetes有一个做好了的服务，也是基于Nginx的，就是Ingress。
==============================================================================
如何访问K8S中的服务：
![image.png]()
1、Ingress介绍
Kubernetes 暴露服务的方式目前只有三种：LoadBlancer Service、NodePort Service、Ingress；前两种估计都应该很熟悉，下面详细的了解下这个 Ingress
Ingress由两部分组成：Ingress Controller 和 Ingress 服务。
Ingress Contronler 通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取它，按照自定义的规则，规则就是写明了哪个域名对应哪个service，生成一段 Nginx 配置，再写到 Nginx-ingress-control的 Pod 里，这个 Ingress Contronler 的pod里面运行着一个nginx服务，控制器会把生成的nginx配置写入/etc/nginx.conf文件中，然后 reload 一下 使用配置生效。以此来达到域名分配置及动态更新的问题。
看个简单的图方便理解：
![image.png]()
ingress控制器有两种：nginx和haproxy 这里是以nginx为讲解。
2、部署一个Nginx Ingress
ingress的部署文件在github Ingress 仓库找到. 针对官方配置我们单独添加了 nodeselector 指定，绑定LB地址 以方便DNS 做解析。
主要用到的文件：
| 1 2 3 4 5 6 7 8 |
$ ls
default-backend.yaml jenkins-ingress.yml nginx-ingress-controller-rbac.yml nginx-ingress-controller.yaml
- - -
default-backend.yaml：这是官方要求必须要给的默认后端，提供404页面的。它还提供了一个http检测功能，检测nginx-ingress-controll健康状态的，通过每隔一定时间访问nginx-ingress-controll的/healthz页面，如是没有响应就
返回404之类的错误码。
nginx-ingress-controller-rbac.yml:这ingress的RBAC授权文件
nginx-ingress-controller.yaml：这是控制器的部署文件。
jenkins-ingress.yml：这是Ingress服务文件，这个可以是任意web程序，里面配置域名与service的对应关系，Ingress称之为规则。
|
第一个是要部署RBAC文件：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 |
cat nginx-ingress-controller-rbac.yml
#apiVersion: v1
#kind: Namespace
#metadata: #这里是创建一个namespace，因为此namespace早有了就不用再创建了
# name: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount #创建一个serveerAcount
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole #这个ServiceAcount所绑定的集群角色
rules:
- apiGroups:
- ""
resources: #此集群角色的权限，它能操作的API资源
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role #这是一个角色，而非集群角色
namespace: kube-system
rules: #角色的权限
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- create
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding #角色绑定
metadata:
name: nginx-ingress-role-nisa-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount #绑定在这个用户
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding #集群绑定
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount #集群绑定到这个serviceacount
namespace: kube-system #集群角色是可以跨namespace，但是这里只指明给这个namespce来使用
|
创建：
| 1 2 3 4 5 6 |
$ kubectl create -f nginx-ingress-controller-rbac.yml
serviceaccount "nginx-ingress-serviceaccount" created
clusterrole "nginx-ingress-clusterrole" created
role "nginx-ingress-role" created
rolebinding "nginx-ingress-role-nisa-binding" created
clusterrolebinding "nginx-ingress-clusterrole-nisa-binding" created
|
RBAC创建完后，就创建default backend服务：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
$ cat default-backend.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: default-http-backend
labels:
k8s-app: default-http-backend
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissable as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: gcr.io/google_containers/defaultbackend:1.0
livenessProbe:
httpGet:
path: /healthz #这个URI是 nginx-ingress-controller中nginx里配置好的localtion
port: 8080
scheme: HTTP
initialDelaySeconds: 30 #30s检测一次/healthz
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
nodeSelector: #指定调度到些Node, 以便后面DNS解析
kubernetes.io/hostname: 10.3.1.17
---
apiVersion: v1
kind: Service #为default backend 创建一个service
metadata:
name: default-http-backend
namespace: kube-system
labels:
k8s-app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
k8s-app: default-http-backend
|
创建：
| 1 2 3 |
$ kubectl create -f default-backend.yaml
deployment "default-http-backend" created
service "default-http-backend" created
|
创建之后查看：
| 1 2 3 4 5 6 7 8 9 10 11 |
root@ubuntu15:/data/ingress# kubectl get rs,pod,svc -n kube-system
NAME DESIRED CURRENT READY AGE
rs/default-http-backend-857b544d94 1 1 1 1m
NAME READY STATUS RESTARTS AGE
po/default-http-backend-857b544d94-bwgjd 1/1 Running 0 1m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/default-http-backend ClusterIP 10.254.208.144 <none> 80/TCP 1m
|
创建好default backend后就要创建nginx-ingress-controller了：
$ cat nginx-ingress-controller.yaml
apiVersion: extensions/v1beta1 kind: Deployment metadata:
name: nginx-ingress-controller
labels:
k8s-app: nginx-ingress-controller
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: nginx-ingress-controller
spec:
# hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used # like with kubeadm # hostNetwork: true #注释表示不使用宿主机的80口，
terminationGracePeriodSeconds: 60
hostNetwork: true #表示容器使用和宿主机一样的网络
serviceAccountName: nginx-ingress-serviceaccount #引用前面创建的serviceacount
containers:
- image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 #容器使用的镜像
name: nginx-ingress-controller #容器名
readinessProbe: #启动这个服务时要验证/healthz 端口10254会在运行的node上监听。
httpGet:
path: /healthz
port: 10254
scheme: HTTP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10 #每隔10做健康检查
timeoutSeconds: 1
ports:
- containerPort: 80
hostPort: 80 #80映射到80
- containerPort: 443
hostPort: 443
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
# - --default-ssl-certificate=$(POD_NAMESPACE)/ingress-secret #这是启用Https时用的
nodeSelector: #指明运行在哪，此IP要和default backend是同一个IP
kubernetes.io/hostname: 10.3.1.17 #上面映射到了hostport80，确保此IP80，443没有占用.
这个控制器就是一个deployment ，里面运行一个容器gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 ，有点像nginx容器，现在创建：
| 1 2 |
$ kubectl create -f nginx-ingress-controller.yaml
deployment "nginx-ingress-controller" created
|
| 1 2 3 4 5 6 7 8 9 10 11 |
root@ubuntu15:/data/ingress# kubectl get rs,pod,svc -n kube-system
NAME DESIRED CURRENT READY AGE
rs/default-http-backend-857b544d94 1 1 1 12m
rs/nginx-ingress-controller-8576d4545d 1 1 0 27s
NAME READY STATUS RESTARTS AGE
po/default-http-backend-857b544d94-bwgjd 1/1 Running 0 12m
po/nginx-ingress-controller-8576d4545d-9tjnv 0/1 ContainerCreating 0 27s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/default-http-backend ClusterIP 10.254.208.144 <none> 80/TCP 12m
|
现在ingress controller 控制器已部署好了，那么如何使用了，那就要写一个ingress规则了，此处就以已存在的jenkins服务为例，配置如何使用域名访问这个service:
| |
$ kubectl get svc,ep
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/jenkinsservice NodePort 10.254.70.47 <none> 8080:30002/TCP 3h
NAME ENDPOINTS AGE
ep/jenkinsservice 172.30.10.15:8080,172.30.11.7:8080 3h
|
现在写个jenkins service的Ingress 规则：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
$ cat jenkins-ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: jenkins-ingress
namespace: default #服务在哪个空间内就写哪个空间
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: ingress.jenkins.com #此service的访问域名
http:
paths:
- backend:
serviceName: jenkinsservice
servicePort: 8080
|
创建它：
$ kubectl create -f jenkins-ingress.yml
ingress "jenkins-ingress" created
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
jenkins-ingress ingress.jenkins.com 80 10s
到这里就已经部署完成了，配置好域名后，就可以用此域名来访问了：
![image.png]()
部署完成了，现在看下nginx-ingress-controller 里nginx配置文件发生了哪些变化：
upstream default-jenkinsservice-8080 {
least_conn;
server 172.30.10.15:8080 max_fails=0 fail_timeout=0;
server 172.30.11.7:8080 max_fails=0 fail_timeout=0;
}
upstream upstream-default-backend {
least_conn;
server 172.30.11.6:8080 max_fails=0 fail_timeout=0;
}
server {
server_name ingress.jenkins.com;
listen [::]:80;
location / {
...
proxy_pass http://default-jenkinsservice-8080;
...
}
}
这些配置都是ingress-controller 自已写入的，动态更新就是它能通过K8S API感知到service的endpoint 发生了变化，然后修改nginx配置并执行reload.
至此，部署完成。
Ingress还有很多部署方式，比如配置https访问的, 以后再写。
本文转自开源中国-kubernetes部署Ingress访问代理与负载均衡器
