使用 docker 后出现的网络异常现象

硬件:

[root@sh-storage-128204 ~]# dmidecode -t system dmidecode 2.12 SMBIOS 2.7 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: To be filled by O.E.M. Product Name: Tecal RH2288H V2-24S Version: V100R002 Serial Number: 2102310QPE10E9000146 UUID: 4A190814-D21D-B211-8DC0-000000821800 Wake-up Type: Power Switch SKU Number: Type1Sku0 Family: Type1Family 

环境:

[root@sh-storage-128204 ~]# docker info Containers: 4 Images: 153 Storage Driver: devicemapper Pool Name: docker-8:17-1075308946-pool Pool Blocksize: 65.54 kB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 3.478 GB Data Space Total: 107.4 GB Data Space Available: 103.9 GB Metadata Space Used: 6.689 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.141 GB Udev Sync Supported: true Deferred Removal Enabled: false Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.10.0-229.el7.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 24 Total Memory: 125.7 GiB Name: sh-storage-128204.sh.vclound.com ID: QV4X:VHQE:EMOI:4TBJ:FZ6K:3N3C:A64Y:PRYR:X2QZ:HHUB:OTND:ZSFF 

用法:
用于搭建 docker private registry
架构:

 nginx <- 本地 docker | |---------------+-------------------| registry registry registry <- 本地 docker |---------------+-------------------| | ceph 集群 (rados) <- 另外一个集群

问题:
搭建了 docker, 启动成功后, 发现上传镜像时偶尔会出现连接卡住问题
检测与 mon 连接发现:

[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200 Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted Offending packet: TCP 10.198.128.204:43301 > 10.198.128.200:6789 S ttl=48 id=11619 iplen=44 seq=2271879518 win=1024 <mss 1460> sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted Offending packet: TCP 10.198.128.204:43302 > 10.198.128.200:6789 S ttl=45 id=9421 iplen=44 seq=2271945055 win=1024 <mss 1460> Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200) Host is up (0.000091s latency). PORT STATE SERVICE 6789/tcp filtered ibm-db2-admin MAC Address: 90:E2:BA:85:21:28 (Intel Corporate) Nmap done: 1 IP address (1 host up) scanned in 0.29 seconds [root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200 Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted Offending packet: TCP 10.198.128.204:48544 > 10.198.128.200:6789 S ttl=54 id=17389 iplen=44 seq=1275261731 win=1024 <mss 1460> sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permitted Offending packet: TCP 10.198.128.204:48545 > 10.198.128.200:6789 S ttl=53 id=40168 iplen=44 seq=1275327266 win=1024 <mss 1460> Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200) Host is up (0.000092s latency). PORT STATE SERVICE 6789/tcp filtered ibm-db2-admin MAC Address: 90:E2:BA:85:21:28 (Intel Corporate) Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds [root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200 Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CST Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200) Host is up (0.00039s latency). PORT STATE SERVICE 6789/tcp open ibm-db2-admin MAC Address: 90:E2:BA:85:21:28 (Intel Corporate) Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds

与 mon 连接时, 会经常出现 6789/tcp filtered ibm-db2-admin 异常现象

另外, 参考系统日志
会出现下面异常 (过滤了大部分, 只显示重要部分信息

/var/log/messages Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet Jan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet .... Jan 20 09:49:44 sh-storage-128204 kernel: net_ratelimit: 137 callbacks suppressed Jan 20 09:49:49 sh-storage-128204 kernel: net_ratelimit: 166 callbacks suppressed Jan 20 09:49:54 sh-storage-128204 kernel: net_ratelimit: 64 callbacks suppressed Jan 20 09:50:01 sh-storage-128204 kernel: net_ratelimit: 132 callbacks suppressed

ping 也会出现异常现场

[root@sh-storage-128204 ~]# ping localhost PING localhost (127.0.0.1) 56(84) bytes of data. ping: sendmsg: Operation not permitted

原因: 当前的网络连接数过大, 导致内核无法响应

解决方法:
修改 /et/sysctl.conf 进行内核参数优化

[root@sh-storage-128204 ~]# sysctl -p vm.swappiness = 10 net.ipv4.ip_forward = 1 net.ipv4.tcp_max_syn_backlog = 8192 net.core.netdev_max_backlog = 8192 net.ipv4.tcp_fin_timeout = 15 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.netfilter.nf_conntrack_max = 1048576 net.nf_conntrack_max = 1048576 net.netfilter.nf_conntrack_tcp_timeout_established = 54000 net.netfilter.nf_conntrack_generic_timeout = 120