Nginx实现动态负载均衡,首先需要一个服务发现集群,通过集群中注册的信息动态更新nginx的配置,实现动态负载均衡。因此首先准备一个Consul集群
Consul集群准备
此处我们仅作为功能架构的测试阶段,因此在一台虚拟机上完成测试,这里准备一台Centos 7.4的虚拟机,IP为192.168.99.12
mkdir /data/consul && cd $_ wget https://releases.hashicorp.com/consul/1.9.3/consul_1.9.3_linux_amd64.zip unzip consul_1.7.3_linux_amd64.zip mv consul /usr/local /bin/
在完成consul的安装之后,需要准备一下集群的基础环境配置
mkdir -pv /data/consul/node{1..3}
我们创建三个consul节点使用的配置文件,分别对应各自的文件夹内
/data/consul/node1/consul_config1.json
{ "datacenter" : "dev" , "data_dir" : "/data/consul/node1" , "log_file" : "/data/consul/node1/consul.log" , "log_level" : "INFO" , "server" : true , "node_name" : "node1" , "ui" : true , "bind_addr" : "192.168.99.12" , "client_addr" : "192.168.99.12" , "advertise_addr" : "192.168.99.12" , "bootstrap_expect" : 3 , "ports" :{ "http" : 8510 , "dns" : 8610 , "server" : 8310 , "serf_lan" : 8311 , "serf_wan" : 8312 } }
/data/consul/node2/consul_config2.json
{ "datacenter" : "dev" , "data_dir" : "/data/consul/node2" , "log_file" : "/data/consul/node2/consul.log" , "log_level" : "INFO" , "server" : true , "node_name" : "node2" , "ui" : true , "bind_addr" : "192.168.99.12" , "client_addr" : "192.168.99.12" , "advertise_addr" : "192.168.99.12" , "bootstrap_expect" : 3, "ports" :{ "http" : 8520, "dns" : 8620, "server" : 8320, "serf_lan" : 8321, "serf_wan" : 8322 } }
/data/consul/node3/consul_config3.json
{ "datacenter" : "dev" , "data_dir" : "/data/consul/node3" , "log_file" : "/data/consul/node3/consul.log" , "log_level" : "INFO" , "server" : true , "node_name" : "node3" , "ui" : true , "bind_addr" : "192.168.99.12" , "client_addr" : "192.168.99.12" , "advertise_addr" : "192.168.99.12" , "bootstrap_expect" : 3 , "ports" :{ "http" : 8530 , "dns" : 8630 , "server" : 8330 , "serf_lan" : 8331 , "serf_wan" : 8332 } }
然后便可以启动consul集群了
nohup consul agent -config-file=/data/consul/node1/consul_config1.json > /dev/null 2>&1 & nohup consul agent -config-file=/data/consul/node2/consul_config2.json -retry-join=192.168.99.12:8311 > /dev/null 2>&1 & nohup consul agent -config-file=/data/consul/node3/consul_config3.json -retry-join=192.168.99.12:8311 > /dev/null 2>&1 &
启动之后,便可以通过地址http://192.168.99.12:8510地址访问,此处192.168.99.12:8510即是Leader角色
consul-cluster-node-dashboard
编译Nginx
做动态负载均衡的时候需要添加nginx-upsync-module和nginx_upstream_check_module两个模块,因此此处我们需要将这两个模块给编译到nginx中去。因为此前我们在基础镜像的时候已经写好了nginx自动化编译的Dockerfile,所以我们这里直接使用即可
Dockerfile ( Nginx 1.14.2)
FROM debian:stretch-slimRUN useradd www && \ mkdir -p /logs/nginx/ /webserver/nginx /webserver/nginx/conf/upsync && \ chown -R www:www /logs/nginx/ /webserver/nginx && \echo 'deb http://mirrors.163.com/debian/ stretch main non-free contrib' > /etc/apt/sources.list && \echo 'deb http://mirrors.163.com/debian/ stretch-updates main non-free contrib' >> /etc/apt/sources.list && \echo 'deb-src http://mirrors.163.com/debian/ stretch main non-free contrib' >> /etc/apt/sources.list && \echo 'deb-src http://mirrors.163.com/debian/ stretch-updates main non-free contrib' >> /etc/apt/sources.list && \echo 'deb-src http://mirrors.163.com/debian/ stretch-backports main non-free contrib' >> /etc/apt/sources.list && \echo 'deb-src http://mirrors.163.com/debian-security/ stretch/updates main non-free contrib' >> /etc/apt/sources.list && \ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ apt-get update && \ apt-get install -y wget vim net-tools unzip libjemalloc-dev && \ apt-get build-dep -y nginx RUN \cd /usr/local /src/ && \ wget -c http://nginx.org/download/nginx-1.14.2.tar.gz && \ wget -c https://www.openssl.org/source /old/1.0.2/openssl-1.0.2m.tar.gz && \ wget -c https://github.com/simplresty/ngx_devel_kit/archive/v0.3.1rc1.tar.gz && \ wget -c https://github.com/openresty/lua-nginx-module/archive/v0.10.11.tar.gz && \ wget -c https://github.com/xiaokai-wang/nginx_upstream_check_module/archive/master.zip -O nginx_upstream_check_module.zip && \ wget -c https://github.com/weibocom/nginx-upsync-module/archive/master.zip -O nginx-upsync-module.zip && \ tar zxf ./nginx-1.14.2.tar.gz && rm nginx-1.14.2.tar.gz && \ tar zxf ./openssl-1.0.2m.tar.gz && rm openssl-1.0.2m.tar.gz && \ tar zxf ./v0.3.1rc1.tar.gz && rm v0.3.1rc1.tar.gz && \ tar zxf ./v0.10.11.tar.gz && rm v0.10.11.tar.gz && \ unzip ./nginx_upstream_check_module.zip && rm nginx_upstream_check_module.zip && \ unzip ./nginx-upsync-module.zip && rm nginx-upsync-module.zip RUN \cd /usr/local /src/nginx-1.14.2 &&\ patch -p1 < /usr/local /src/nginx_upstream_check_module-master/check_1.12.1+.patch &&\ ./configure \ --prefix=/webserver/nginx \ --user=www --group=www --with-pcre \ --with-stream \ --with-http_v2_module \ --with-http_ssl_module \ --with-ld-opt=-ljemalloc \ --with-http_realip_module \ --with-http_gzip_static_module \ --with-http_stub_status_module \ --http-log-path=/logs/nginx/access.log \ --error-log-path=/logs/nginx/error.log \ --with-openssl=/usr/local /src/openssl-1.0.2m \ --add-module=/usr/local /src/ngx_devel_kit-0.3.1rc1 \ --add-module=/usr/local /src/lua-nginx-module-0.10.11 \ --add-module=/usr/local /src/nginx_upstream_check_module-master \ --add -module=/usr/local /src/nginx-upsync-module-master && \ make && \ make install
upstream测试服务
此处我们可以写个http的demo服务也行,或者更简单点,也可以直接使用Docker运行两个服务.此处我们向consul集群中注册两个已有的服务
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10, "down":0}' http://192.168.99.12:8510/v1/kv/upstreams/app/192.168.99.12:9000 curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10, "down":0}' http://192.168.99.12:8510/v1/kv/upstreams/app/192.168.99.12:9001
执行完之后在Dashboard中的k/v处即可看到
consul cluster key/value
准备Nginx动态更新的配置文件
我们这里作为功能测试实验,所以准备一个简洁的nignx.conf配置文件
worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; upstream app { upsync 192.168.99.12:8510/v1/kv/upstreams/app/ upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off; upsync_dump_path /webserver/nginx/conf/app.conf; # 当consul故障时候,就可以把此作为备份配置文件 include /webserver/nginx/conf/app.conf; # 准备一个兼容的nginx测试文件,如果没有第一次启动可能起不来 check interval=1000 rise=2 fall=2 timeout=3000 type=http default_down=false; check_http_send "HEAD / HTTP/1.0\r\n\r\n"; check_http_expect_alive http_2xx http_3xx; } server { listen 80; server_name localhost; location / { proxy_pass http://app; } location /upstream_list { upstream_show; } location /upstream_status { check_status; access_log off; } } }
server 127.0.0.1:80 weight=1 max_fails=2 fail_timeout=10s down; server 127.0.0.1:81 weight=1 max_fails=2 fail_timeout=10s down;
准备完成之后,我们就可以启动nginx服务了
docker run -d --name nginx-consul -v /Users/marionxue/Downloads/webserver/nginx.conf:/webserver/nginx/conf/nginx.conf -v /Users/marionxue/Downloads/webserver/app.conf:/webserver/nginx/conf/app.conf -p 80:80 -p 443:443 nginx:upsync
容器运行正常后,就可以通过http://127.0.0.1/upstream_list查看upstream主机了,此时app.conf的内容也会被动态的更新,这里可以作为consul故障后的备份配置文件
容器启动后的app.conf的内容
root@be1b245c1c47:/webserver/nginx/sbin# cat ../conf/app.conf server 192.168.99.12:9001 weight=1 max_fails=2 fail_timeout=10s; server 192.168.99.12:9000 weight=1 max_fails=2 fail_timeout=10s;
consul cluster upstream list
此处通过curl增加一个新的主机节点观察是否会自动的更新
> curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10, "down":1}' http://192.168.99.12:8510/v1/kv/upstreams/app/192.168.99.12:8510true
增加新的主机节点
这和我们预期的一样,不需要干预nginx服务,即可自动完成nginx配置的更新。此处即基本上完成nginx实现动态负载均衡的基础实验。