Sysdig绝对是可以跟SystemTap相媲美的工具，都是系统故障排查的利器，不论是开发人员还是运维人员都必须要掌握的一个工具。今天先介绍一下Sysdig的一些特性。官网上说Sysdig是Strace, Tcpdump, Lsof, Htop, Iftop等工具的合体，至于你信不信只有使用之后才能来惊叹我说的到底对不对。这里官网总结了sysdig命令对于strace, tcpdump, lsof, iftop的对比：https://sysdig.com/blog/linux-troubleshooting-cheatsheet/。所以Sysdig主要功能就是系统级别的探测，捕捉服务器上的应用程序对CPU, Memory, I/O, Network的状况。
Sysdig最新版提供了Docker容器镜像，可以很方便的直接拉取Docker镜像，另一方它提供容器级别的信息采集指令（sysdig -pc container.name=your_container_name），支持查询指定容器之间的网络流量、指定容器的CPU使用率等。
公司旗下的商用软件Sysdig Cloud则是容器级别的系统信息和网络流量监控、调试软件，这个在CoreOS Fest 大会上有介绍，它支持Real-Time Dashboard, Historical Replay, Dynamic Topology and Intelligent Alert, 可以想象成Nagios对系统的监控。
软件安装请参考官方文档：http://www.sysdig.org/install/ 相对于SystemTap的安装Sysdig更容易些，本篇文章有点长就不浪费在安装上了，熟悉Ansible的可以去直接用sysdig的Galaxy：https://galaxy.ansible.com/detail#/role/692
Sysdig的语法在record 和replay系统跟踪方面跟Tcpdump和perf很像；在系统性能分析方面的语法chisels又跟SystemTap和dstat的--top*很像，只不过SystemTap需要自己写tap（代码写好了，比Sysdig强大）, Sysdig是已经帮你写好了；在交互式使用方面又跟htop很像。
不废话了，下面介绍一下基本的使用方法。
一，基础功能介绍：
最简单的使用方法是直接输入sysdig, 他会捕获系统的每一个事件并且直接输出到屏幕。
![wKioL1chxa2ivZPEAAdCNLgHDUQ609.jpg]( "1.jpg")
每一个event都是一行信息，输出的格式如下：
*%evt.num %evt.time %evt.cpu %proc.name (%thread.tid) %evt.dir %evt.type %evt.args
evt.num 是一个自动增长的event number
evt.time 是时间戳
evt.cpu 是捕获到的在用的哪颗CPU
proc.name 是process name
thread.tid 是thread id,如果不是多线程，就是pid
evt.dir 是event方向，>代表进入events,<代表退出events
evt.type 是event的操作名称，e.g. open or read
evt.args 是event 调用命令参数的列表
如果要保存信息的话，可以加上-w选项,限制捕捉行数用-n：
sysdig -n 100 -w firstcapture.scap
如果是按大小区分捕获文件的话，可以用-C 1 默认捕获1M文件就分割到下一个文件，-W 5 参数只捕获5个文件
sysdig -C 1 -W 5 -w secondcapture.scap
生成的文件名是secondcapture.scap.{1..5}。
要阅读刚才捕获的文件，用-r参数：
sysdig -r firstcapture.scap
如果要对捕获的内容做具体的事件查询呢？请看下面
查询进程是mysqld的event：
sysdig -r firstcapture.scap proc.name=mysqld
![wKiom1chzx_Co3yPAACbEqTMWH4253.png]( "2.png")
看proc.name 是mysql 或者nginx的，并且event是read的event
sysdig -r firstcapture.scap proc.name=mysqld or proc.name=nginx and env.type=read
筛选的关键字可以用sysdig -l 来获得，他支持以下12中Field Class：
fd
process
evt
user
group
syslog
container
fdlist
k8s
mesos
span
evtin
# sysdig -l
----------------------
Field Class: fd
fd.num the unique number identifying the file descriptor.
fd.type type of FD. Can be 'file', 'directory', 'ipv4', 'ipv6', 'unix',
'pipe', 'event', 'signalfd', 'eventpoll', 'inotify' or 'signal
fd'.
fd.typechar type of FD as a single character. Can be 'f' for file, 4 for IP
v4 socket, 6 for IPv6 socket, 'u' for unix socket, p for pipe,
'e' for eventfd, 's' for signalfd, 'l' for eventpoll, 'i' for i
notify, 'o' for uknown.
fd.name FD full name. If the fd is a file, this field contains the full
path. If the FD is a socket, this field contain the connection
tuple.
----------------------
Field Class: process
proc.pid the id of the process generating the event.
proc.exe the first command line argument (usually the executable name or
a custom one).
proc.name the name (excluding the path) of the executable generating the
event.
proc.args the arguments passed on the command line when starting the proc
ess generating the event.
proc.env the environment variables of the process generating the event.
proc.cmdline full process command line, i.e. proc.name + proc.args.
proc.exeline full process command line, with exe as first argument, i.e. pro
c.exe + proc.args.
proc.cwd the current working directory of the event.
proc.nthreads the number of threads that the process generating the event cur
rently has, including the main process thread.
proc.nchilds the number of child threads that the process generating the eve
nt currently has. This excludes the main process thread.
----------------------
Field Class: evt
evt.num event number.
evt.time event timestamp as a time string that includes the nanosecond p
art.
evt.time.s event timestamp as a time string with no nanoseconds.
evt.datetime event timestamp as a time string that includes the date.
evt.rawtime absolute event timestamp, i.e. nanoseconds from epoch.
evt.rawtime.s integer part of the event timestamp (e.g. seconds since epoch).
evt.rawtime.ns fractional part of the absolute event timestamp.
evt.reltime number of nanoseconds from the beginning of the capture.
evt.reltime.s number of seconds from the beginning of the capture.
evt.reltime.ns fractional part (in ns) of the time from the beginning of the c
apture.
evt.latency delta between an exit event and the correspondent enter event,
in nanoseconds.
----------------------
Field Class: user
user.uid user ID.
user.name user name.
user.homedir home directory of the user.
user.shell user's shell.
----------------------
Field Class: group
group.gid group ID.
group.name group name.
----------------------
Field Class: syslog
syslog.facility.str
facility as a string.
syslog.facility facility as a number (0-23).
syslog.severity.str
severity as a string. Can have one of these values: emerg, aler
t, crit, err, warn, notice, info, debug
syslog.severity severity as a number (0-7).
syslog.message message sent to syslog.
----------------------
Field Class: container
container.id the container id.
container.name the container name.
container.p_w_picpath the container p_w_picpath.
container.type the container type, eg: docker or rkt
----------------------
Field Class: fdlist
fdlist.nums for poll events, this is a comma-separated list of the FD numbe
rs in the 'fds' argument, returned as a string.
fdlist.names for poll events, this is a comma-separated list of the FD names
in the 'fds' argument, returned as a string.
fdlist.cips for poll events, this is a comma-separated list of the client I
P addresses in the 'fds' argument, returned as a string.
fdlist.sips for poll events, this is a comma-separated list of the server I
P addresses in the 'fds' argument, returned as a string.
fdlist.cports for TCP/UDP FDs, for poll events, this is a comma-separated lis
t of the client TCP/UDP ports in the 'fds' argument, returned a
s a string.
fdlist.sports for poll events, this is a comma-separated list of the server T
CP/UDP ports in the 'fds' argument, returned as a string.
----------------------
Field Class: k8s
k8s.pod.name Kubernetes pod name.
k8s.pod.id Kubernetes pod id.
k8s.pod.label Kubernetes pod label. E.g. 'k8s.pod.label.foo'.
k8s.pod.labels Kubernetes pod comma-separated key/value labels. E.g. 'foo1:bar
1,foo2:bar2'.
k8s.rc.name Kubernetes replication controller name.
k8s.rc.id Kubernetes replication controller id.
k8s.rc.label Kubernetes replication controller label. E.g. 'k8s.rc.label.foo
'.
k8s.rc.labels Kubernetes replication controller comma-separated key/value lab
els. E.g. 'foo1:bar1,foo2:bar2'.
k8s.svc.name Kubernetes service name (can return more than one value, concat
enated).
k8s.svc.id Kubernetes service id (can return more than one value, concaten
ated).
----------------------
Field Class: mesos
mesos.task.name Mesos task name.
mesos.task.id Mesos task id.
mesos.task.label
Mesos task label. E.g. 'mesos.task.label.foo'.
mesos.task.labels
Mesos task comma-separated key/value labels. E.g. 'foo1:bar1,fo
o2:bar2'.
mesos.framework.name
Mesos framework name.
mesos.framework.id
Mesos framework id.
marathon.app.name
Marathon app name.
marathon.app.id Marathon app id.
marathon.app.label
Marathon app label. E.g. 'marathon.app.label.foo'.
----------------------
Field Class: span
span.id tracer ID. This is a unique identifier that is used to match th
e enter and exit tracer events for this span. It can also be us
ed to match different spans belonging to a trace.
span.time time of the span enter tracer as a time string that includes th
e nanosecond part.
span.ntags number of tags that this span has.
span.nargs number of arguments that this span has.
span.tags dot-separated list of the span's tags.
span.tag one of the span's tags, specified by 0-based offset, e.g. 'span
.tag[1]'. You can use a negative offset to pick elements from t
he end of the tag list. For example, 'span.tag[-1]' returns the
last tag.
----------------------
Field Class: evt
evtin.span.id (FILTER ONLY) the ID of the trace span containing the event.
evtin.span.ntags
(FILTER ONLY) the number of tags of the trace span containing t
he event.
evtin.span.nargs
(FILTER ONLY) the number of arguments of the trace span contain
ing the event.
evtin.span.tags (FILTER ONLY) the comma-separated list of tags of the trace spa
n containing the event.
evtin.span.tag (FILTER ONLY) one of the tags of the trace span containing the
event, specified by offset. E.g. 'evtin.span.tag[1]'. You can u
se a negative offset to pick elements from the end of the tag l
ist. For example, 'evtin.span.tag[-1]' returns the last tag.
如果不想要sysdig默认的输出格式，也可以自己定制，类似于top的定制。变量名必须用%开头。
# sysdig -r sysdigdump.sap -p "user:%user.name dir:%evt.arg.path" evt.type=chdir
user:root dir:/
user:root dir:/
输出一个程序正在被哪个用户使用，被调用的命令参数是怎样的：
# sysdig -p"%user.name) %proc.name %proc.args" evt.type=execve and evt.arg.ptid=zsh
root) git config --get oh-my-zsh.hide-status
root) git symbolic-ref HEAD
root) git rev-parse --short HEAD
root) ls --color=tty --color=auto -hltr
root) git config --get oh-my-zsh.hide-status
root) git symbolic-ref HEAD
root) git rev-parse --short HEAD
可以我在zsh的命令行上执行了ls 命令， 并且oh-my-zsh在后台悄悄地执行git config 命令，没有sysdig工具你是没法知道oh-my-zsh在没执行一次命令都在后台执行git config的，果断把oh-my-zsh自动更新给禁掉，以后可以每隔一段时间手动去更新oh-my-zsh。
查看哪个进程在/etc下写文件，同样效果lsof也可以实现，只是显示不如sysdig，而且sysdig还可以定制显示内容。
# root at shanker in ~/.oh-my-zsh on git:master o [9:22:59]
# sysdig -p "user:%user.name process:%proc.name file:%fd.name" "evt.type=write and fd.name contains /etc"
user:root process:vim file:/etc/ansible/.hosts.swp
user:root process:vim file:/etc/ansible/.hosts.swp
user:root process:vim file:/etc/ansible/.hosts.swp
^C#
# root at shanker in ~/.oh-my-zsh on git:master o [9:27:32]
# lsof -cv /etc
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
vmstat 59 root cwd DIR 202,1 4096 2 /
vmstat 59 root rtd DIR 202,1 4096 2 /
vmstat 59 root txt unknown /proc/59/exe
vim 11547 root cwd DIR 202,1 4096 16386 /root
vim 11547 root rtd DIR 202,1 4096 2 /
vim 11547 root txt REG 202,1 2191736 1026 /usr/bin/vim.basic
vim 11547 root mem REG 202,1 47712 418583 /lib/x86_64-linux-gnu/libnss_files-2.19.so
vim 11547 root mem REG 202,1 47760 418575 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
vim 11547 root mem REG 202,1 97296 418568 /lib/x86_64-linux-gnu/libnsl-2.19.so
vim 11547 root mem REG 202,1 39824 418566 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
vim 11547 root mem REG 202,1 10680 418582 /lib/x86_64-linux-gnu/libutil-2.19.so
vim 11547 root mem REG 202,1 100728 396113 /lib/x86_64-linux-gnu/libz.so.1.2.8
vim 11547 root mem REG 202,1 18624 396035 /lib/x86_64-linux-gnu/libattr.so.1.1.0
vim 11547 root mem REG 202,1 252032 397658 /lib/x86_64-linux-gnu/libpcre.so.3.13.1
vim 11547 root mem REG 202,1 1840928 418580 /lib/x86_64-linux-gnu/libc-2.19.so
vim 11547 root mem REG 202,1 141574 418570 /lib/x86_64-linux-gnu/libpthread-2.19.so
vim 11547 root mem REG 202,1 3480880 895 /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
vim 11547 root mem REG 202,1 14664 418562 /lib/x86_64-linux-gnu/libdl-2.19.so
vim 11547 root mem REG 202,1 27080 8084 /usr/lib/x86_64-linux-gnu/libgpm.so.2
vim 11547 root mem REG 202,1 31168 396033 /lib/x86_64-linux-gnu/libacl.so.1.1.0
vim 11547 root mem REG 202,1 134296 396097 /lib/x86_64-linux-gnu/libselinux.so.1
vim 11547 root mem REG 202,1 167096 396105 /lib/x86_64-linux-gnu/libtinfo.so.5.9
vim 11547 root mem REG 202,1 1071552 418560 /lib/x86_64-linux-gnu/libm-2.19.so
vim 11547 root mem REG 202,1 149120 418572 /lib/x86_64-linux-gnu/ld-2.19.so
vim 11547 root mem REG 202,1 1607664 1578 /usr/lib/locale/locale-archive
vim 11547 root 0u CHR 136,7 0t0 10 /dev/pts/7
vim 11547 root 1u CHR 136,7 0t0 10 /dev/pts/7
vim 11547 root 2u CHR 136,7 0t0 10 /dev/pts/7
vim 11547 root 4u REG 202,1 12288 131619 /etc/ansible/.hosts.swp
想要看网络连接，使用文件描述法是ipv4即可：
sysdig fd.type=ipv4 and evt.type=write
如果使用container的情况下，输出格式需要重新格式化才能友好阅读，根据自己container的类型选择命令参数：
Using -pc or -pcontainer, the default format will be changed to a container-friendly one:
%evt.num %evt.outputtime %evt.cpu %container.name (%container.id) %proc.name (%thread.tid:%thread.vtid) %evt.dir %evt.type %evt.info
Using -pk or -pkubernetes, the default format will be changed to a kubernetes-friendly one:
%evt.num %evt.outputtime %evt.cpu %k8s.pod.name (%container.id) %proc.name (%thread.tid:%thread.vtid) %evt.dir %evt.type %evt.info
Using -pm or -pmesos, the default format will be changed to a mesos-friendly one:
%evt.num %evt.outputtime %evt.cpu %mesos.task.name (%container.id) %proc.name (%thread.tid:%thread.vtid) %evt.dir %evt.type %evt.info
二，Chisels用法简介：
因为sysdig的图标是铲子，这个Chisel字面翻译是凿子，铲子加上凿子可以帮你挖的更深。
用过SystemTap（Dtrace Solaris的叫法）工具的朋友一定熟悉他的stap脚本，这个脚本语言需要单独去学习，使用成本比较高，sysdig呢直接提供了很多chisels共大家使用，免去了自己开发脚本的时间，它是用Lua脚本语言开发的。他俩的功能都是挖掘系统层的调用事件，挖的很深很深......
引用一下官网的“自吹”：
Sysdig’s chisels are little scripts that analyze the sysdig event stream to perform useful actions. If you’ve used system tracing tools like dtrace, you’re probably familiar with running scripts that trace OS events. Usually, with dtrace-like tools you write your scripts using a domain-specific language that gets compiled into bytecode and injected in the kernel. Sysdig uses a different approach: events are efficiently brought to user-level, enriched with context, and then scripts can be applied to them. This brings several benefits:
A well known scripting language can be used instead of a custom one. In fact, sysdig’s chisels are Lua scripts. Lua is well known, powerful, stable and extremely efficient.
Chisels can leverage the broad collection of Lua libraries.
Chisels work well on live systems, but can also be used with capture files for offline analysis.
下面是一个SystemTap的stap脚本，这个工具会单独开一篇文章来介绍，也是史诗级的。
# cat nginx.stap
global s;
global quit = 0;
probe timer.profile {
if (pid() == target()) {
if (quit) {
foreach (i in s-) {
print_ustack(i);
printf("\t%d\n", @count(s[i]));
}
exit()
} else {
s[ubacktrace()] <<< 1;
}
}
}
probe timer.s(20) {
quit = 1
}
要使用sysdig的chisels, 用-cl参数来查看，如果想对其中的某一个感兴趣，使用-i来详细查看，sysdig源码在github上，您也可以自己去创造自己的chisel。
# sysdig -cl
Category: Application
---------------------
httplog HTTP requests log
httptop Top HTTP requests
memcachelog memcached requests log
Category: CPU Usage
-------------------
spectrogram Visualize OS latency in real time.
subsecoffset Visualize subsecond offset execution time.
topcontainers_cpu
Top containers by CPU usage
topprocs_cpu Top processes by CPU usage
Category: Errors
----------------
topcontainers_error
Top containers by number of errors
topfiles_errors Top files by number of errors
topprocs_errors top processes by number of errors
Category: I/O
-------------
echo_fds Print the data read and written by processes.
.....
# root at shanker in ~ [1:52:57]
# sysdig -i echo_fds
Category: I/O
-------------
echo_fds Print the data read and written by processes.
Print the data read and written for any FD. Combine this script with a filter t
o restrict what it shows. This chisel is compatible with containers using the s
ysdig -pc or -pcontainer argument, otherwise no container information will be s
hown. (Blue represents [Write], and Red represents [Read] for all data except
when the -pc or -pcontainer argument is used. If used the container.name and co
ntainer.id will be represented as: Green [host], and Cyan [container]) Containe
r information will contain '[]' around container.name and container.id.
Args:
[string] disable_color - Set to 'disable_colors' if you want to
disable color output
如果要运行某一个chisel，直接跟-c即可，比如查看top process by CPU, Network I/O, File I/O：
sysdig -c topprocs_cpu{topprocs_file,topprocs_net}
Bytes Process PID
--------------------------------------------------------------------------------
488B sshd 18115
如何查询一个特定进程的所有网络流量呢？拿nginx为例，以ASCII字符显示nginx进程的所有网络流量，回显蓝色字体是写，红色字体是read：
# root at shanker in ~ [2:00:35]
# sysdig -A -c echo_fds proc.name=nginx
------ Read 73B from 127.0.0.1:54811->127.0.0.1:http (nginx)
GET / HTTP/1.1
User-Agent: curl/7.35.0
Host: localhost
Accept: */*
------ Write 504B to ffff880007b07080->ffff880007b05640 /var/run/php5-fpm.sock (nginx)
QUERY_STRINGREQUEST_METHODGETCONTENT_TYPECONTENT_LENGTHSCRIPT_FILENAME/var/www/html/index.php
SCRIPT_NAME/index.phpREQUEST_URI/
DOCUMENT_URI/index.php
DOCUMENT_ROOT/var/www/htmlSERVER_PROTOCOLHTTP/1.1GATEWAY_INTERFACECGI/1.1SERVER_SOFTWAREnginx/1.4.6REMOTE_ADDR127.0.0.1REMOTE_PORT54811SERVER_ADDR127.0.0.1SERVER_PORT80SERVER_NAMEshanker.heyoa.comREDIRECT_STATUS200HTTP_USER_AGENTcurl/7.35.0HTTP_HOSTlocalhostHTTP_ACCEPT*/*
------ Read 4.00KB from ffff880007b07080->ffff880007b05640 /var/run/php5-fpm.sock (nginx)
X-Powered-By: PHP/5.5.9-1ubuntu4.16
Content-Type: text/html; charset=UTF-8
如果把proc.name=nginx换成fd.sport=80是一样的效果。
如果想查看sshd进程交互的，server ip 是172.31.23.95的流量：
sysdig -A -s 65000 -n 1000 -c echo_fds fd.sip=172.31.23.95 and proc.name=sshd
我们在拿官网的topfiles_bytes chisel来学习一下。如果不接任何参数就显示的就是读写磁盘字节数最高的那些进程：
![wKioL1ciyZGAXAepAAA7TMBEgG4687.png]( "3.png")
$ sysdig -c topfiles_bytes
Bytes Filename
------------------------------
23.32KB /proc/net/unix
9.11KB /usr/share/icons/hicolor/16x16/actions/terminator_receive_off.png
5.64KB /etc/localtime
4.92KB /proc/interrupts
4.38KB /dev/input/event2
4.37KB /etc/wgetrc
2.88KB /proc/stat
2.39KB /usr/share/locale/locale.alias
1.85KB /proc/18263/status
如果不想看访问/dev的进程，可以这样：
sysdig -c topfiles_bytes "not fd.name contains /dev"
如果只想看/root目录下读写比较高的进程，可以这样：
$ sysdig -c topfiles_bytes "fd.name contains /root"
Bytes Filename
------------------------------
1.29KB /root/agent/build/debug/test/index.html.93
1.10KB /root/.dropbox/PENDING_aWX7WU
1.10KB /root/.dropbox/UPDATED_UsxrsX
16B /root/.dropbox/filecache.dbx```
如果想看某个进程对文件的读写，还可以这样：
$ sysdig -c topfiles_bytes "proc.name=vi"
Bytes Filename
------------------------------
4.00KB /root/agent/build/debug/test/.lo.txt.swp
3.36KB /usr/share/terminfo/x/xterm-256color
2.18KB /etc/passwd
1.98KB /root/agent/build/debug/test/lo.txt
1.92KB /etc/virc
1.66KB /etc/nsswitch.conf
832B /lib64/libpcre.so.1
832B /lib64/libc.so.6
832B /lib64/libnss_files.so.2
或者针对某个用户的：
$ sysdig -c topfiles_bytes "user.name=loris"
Bytes Filename
------------------------------
3.31KB /etc/nsswitch.conf
2.18KB /etc/passwd
1.62KB /lib64/libselinux.so.1
1.62KB /lib64/libc.so.6
1.62KB /lib64/libpcre.so.1
1.62KB /lib64/libdl.so.2
1.62KB /lib64/libnss_files.so.2
898B /etc/group
54B /proc/self/task/30414/attr/current
三，csysdig简介
csysdig的用法就有点类似htop了，是sysdig交互式的用户界面，在熟悉了sysdig的用法的基础上，用csysdig就轻车熟路了，在这里就不详细介绍了，大家可以参考官方文档http://www.sysdig.org/wiki/csysdig-overview/
https://github.com/draios/sysdig/blob/dev/userspace/sysdig/man/csysdig.md
和youtube上的视频介绍：
https://www.youtube.com/watch?v=UJ4wVrbP-Q8
最后总结一下，sysdig真的是一款非常强大的系统分析工具，本文只是抛砖引玉，希望大家多多挖掘，分享更深层的东西来分析服务器的瓶颈和故障。
参考连接：
http://www.sysdig.org/wiki/sysdig-user-guide/
http://www.sysdig.org/wiki/chisels-user-guide/
http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
http://xmodulo.com/monitor-troubleshoot-linux-server-sysdig.html
