排查过很多次pod网络有问题的场景 ，一直没太弄明白，pod内的网络报文怎么抓，今天遇到一个liveness健康检查失败的问题，liveness是kubelet去访问pod（发生get请求）来确认的，那么对应的访问日志在pod内是可以看到的，因此可以尝试抓包排查，但是pod并不能简简单单的使用tcpdump抓包即可，docker/kubectl cp 一个tcpdump进去也不好使，那么，pod内的包怎么抓呢？
容器的网络隔离使用的是linux的network namespace ，因此我们到对应的ns里面抓包即可，下面演示
1.查看指定 pod 运行在哪个宿主机上
拿到宿主机的信息，然后登陆上去 cn-shenzhen.192.168.0.178

# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE busybox-5fc755d7d9-nc8bz 1/1 Running 2 5d5h 172.20.2.21 cn-shenzhen.192.168.0.178 <none>

2.获得容器的 pid
登录到对应的node上，下面演示的是两种找pid的方法

# docker ps|grep busy d5ae39bad811 busybox "sleep 360000" 29 minutes ago Up 29 minutes k8s_busybox_busybox-5fc755d7d9-nc8bz_default_b9a845f1-f09b-11e9-a7ea-00163e0e34c8_2 cdab20715cd9 registry-vpc.cn-shenzhen.aliyuncs.com/acs/pause-amd64:3.0 "/pause" 29 minutes ago Up 29 minutes k8s_POD_busybox-5fc755d7d9-nc8bz_default_b9a845f1-f09b-11e9-a7ea-00163e0e34c8_4 [root@izwz9314kt10onbwuw6odez ~]# docker inspect -f {{.State.Pid}} d5ae39bad811 6875

3.进入该容器的 network namespace

nsenter依赖util-linux，因此提前安装

# yum -y install util-linux.x86_64

进入到对应容器的network ns里面，并指向ip a查看ip

 # nsenter --target 6875 -n [root@izwz9314kt10onbwuw6odez ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP link/ether d2:cc:e9:b1:f6:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.20.2.21/25 brd 172.20.2.127 scope global eth0 valid_lft forever preferred_lft forever 对比一下第一步拿到的pod ip 是一样的

4.使用 tcpdump 抓包，指定 eth0 网卡
直接抓包尝试

tcpdump -i eth0 tcp and port 80 -vvv 新开一个窗口进入到pod里面配合访问测试 # kubectl exec -it busybox-5fc755d7d9-nc8bz sh / # nc -vz www.baidu.com 80 www.baidu.com (14.215.177.38:80) open

5.退出network namespace
直接exit即可

# exit logout [root@izwz9314kt10onbwuw6odez ~]# ifconfig cali4f23c4d62c6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.20.2.1 netmask 255.255.255.255 broadcast 172.20.2.1 ether 3e:65:de:6b:77:69 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ......

实战：模拟kubelet的liveness健康检查失败抓包
1.起个nginx的deployment，加上一个不存在的路径做liveness

livenessProbe: failureThreshold: 5 httpGet: path: /muyuan.html port: 80 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 20 successThreshold: 1 timeoutSeconds: 1

2，确认node，并登陆获取pid

# kubectl get pods -o wide -l app=nginx NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE nginx-deployment-basic-58d68cc69d-cg9xr 1/1 Running 0 10s 172.20.2.233 cn-shenzhen.192.168.0.131 <none> 获取pid # docker inspect -f {{.State.Pid}} ca32f2599302 7030 进入ns # nsenter --target 7030 -n

3,抓到kubelet的包了

# tcpdump -i any port 80 -nnvv tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 18:33:15.996389 IP (tos 0x0, ttl 64, id 9563, offset 0, flags [DF], proto TCP (6), length 60) 172.20.2.129.34098 > 172.20.2.233.80: Flags [S], cksum 0x5dc1 (incorrect -> 0x3346), seq 2310580335, win 29200, options [mss 1460,sackOK,TS val 3575092 ecr 0,nop,wscale 9], length 0 18:33:15.996426 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 172.20.2.233.80 > 172.20.2.129.34098: Flags [S.], cksum 0x5dc1 (incorrect -> 0xdef8), seq 2455778657, ack 2310580336, win 28960, options [mss 1460,sackOK,TS val 3575092 ecr 3575092,nop,wscale 9], length 0 18:33:15.996448 IP (tos 0x0, ttl 64, id 9564, offset 0, flags [DF], proto TCP (6), length 52) 172.20.2.129.34098 > 172.20.2.233.80: Flags [.], cksum 0x5db9 (incorrect -> 0x7ead), seq 1, ack 1, win 58, options [nop,nop,TS val 3575092 ecr 3575092], length 0 18:33:15.996591 IP (tos 0x0, ttl 64, id 9565, offset 0, flags [DF], proto TCP (6), length 176) 172.20.2.129.34098 > 172.20.2.233.80: Flags [P.], cksum 0x5e35 (incorrect -> 0x3eab), seq 1:125, ack 1, win 58, options [nop,nop,TS val 3575092 ecr 3575092], length 124: HTTP, length: 124 GET /muyuan.html HTTP/1.1 Host: 172.20.2.233:80 User-Agent: kube-probe/1.12+ Accept-Encoding: gzip Connection: close 18:33:15.996611 IP (tos 0x0, ttl 64, id 47190, offset 0, flags [DF], proto TCP (6), length 52) 172.20.2.233.80 > 172.20.2.129.34098: Flags [.], cksum 0x5db9 (incorrect -> 0x7e32), seq 1, ack 125, win 57, options [nop,nop,TS val 3575092 ecr 3575092], length 0 18:33:15.996721 IP (tos 0x0, ttl 64, id 47191, offset 0, flags [DF], proto TCP (6), length 369) 172.20.2.233.80 > 172.20.2.129.34098: Flags [P.], cksum 0x5ef6 (incorrect -> 0xe1af), seq 1:318, ack 125, win 57, options [nop,nop,TS val 3575092 ecr 3575092], length 317: HTTP, length: 317 HTTP/1.1 404 Not Found Server: nginx/1.7.9 Date: Tue, 22 Oct 2019 10:33:15 GMT Content-Type: text/html Content-Length: 168 Connection: close <html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.7.9</center> </body> </html> 18:33:15.996747 IP (tos 0x0, ttl 64, id 9566, offset 0, flags [DF], proto TCP (6), length 52)