观测工具下载地址：https://github.com/iovisor/bcc
bcc观测工具的安装方式：https://github.com/iovisor/bcc/blob/master/INSTALL.md
工具简介：
BCC makes BPF programs easier to write, with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua. It is suited for many tasks, including performance analysis and network traffic control.
BCC 是用于创建高效内核跟踪和操作程序的工具包，包括一些有用的工具和示例。它利用了扩展的BPF（Berkeley Packet Filters），正式名称为 eBPF。
使用要求：
1）Linux 内核版本最低 4.1
2）kernel-devel 版本必须和内核版本一致
[root@yang-01 ~]# rpm -qa|grep -i kernel | grep -i devel
kernel-devel-3.10.0-1127.18.2.el7.x86_64
[root@yang-01 ~]# uname -r
3.10.0-1127.el7.x86_64
3）kernel 配置文件相关参数必须按照下面的要求
[root@yang-01 ~]# grep -i BPF /boot/config-`uname -r`
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
4）对于 CentOS 建议升级到 3.6 及以上版本
5）使用默认的 Python 2.7 即可，不需要升级 Python 3
yum安装
[root@yang-01 ~]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
[root@yang-01 ~]# uname -r
3.10.0-1127.el7.x86_64
[root@yang-01 ~]# yum update kernel
Loaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies--> Running transaction check---> Package kernel.x86_64 0:3.10.0-1127.18.2.el7 will be installed
--> Finished Dependency ResolutionDependencies Resolved=========================================================================================================================================================================== Package Arch Version Repository Size
===========================================================================================================================================================================Installing: kernel x86_64 3.10.0-1127.18.2.el7 updates 50 M
Transaction Summary===========================================================================================================================================================================Install 1 Package
Total download size: 50 M
Installed size: 64 M
Is this ok [y/d/N]: y
Downloading packages:Delta RPMs disabled because /usr/bin/applydeltarpm not installed.kernel-3.10.0-1127.18.2.el7.x86_64.rpm | 50 MB 00:00:08 Running transaction checkRunning transaction testTransaction test succeededRunning transaction Installing : kernel-3.10.0-1127.18.2.el7.x86_64 1/1
Verifying : kernel-3.10.0-1127.18.2.el7.x86_64 1/1
Installed: kernel.x86_64 0:3.10.0-1127.18.2.el7
Complete![root@yang-01 ~]# yum install bcc -y
Loaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies--> Running transaction check---> Package bcc.x86_64 0:0.10.0-1.el7 will be installed
--> Processing Dependency: bcc-tools = 0.10.0-1.el7 for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: llvm-private >= 6.0.1-0.3 for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libLLVM-7-rhel.so(LLVM_7)(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangSerialization.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangSema.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangRewrite.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangParse.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangLex.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangFrontend.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangEdit.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangDriver.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangCodeGen.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangBasic.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangAnalysis.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libclangAST.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Processing Dependency: libLLVM-7-rhel.so()(64bit) for package: bcc-0.10.0-1.el7.x86_64
--> Running transaction check---> Package bcc-tools.x86_64 0:0.10.0-1.el7 will be installed
--> Processing Dependency: python-bcc = 0.10.0-1.el7 for package: bcc-tools-0.10.0-1.el7.x86_64
--> Processing Dependency: python-netaddr for package: bcc-tools-0.10.0-1.el7.x86_64
--> Processing Dependency: kernel-devel for package: bcc-tools-0.10.0-1.el7.x86_64
---> Package llvm-private.x86_64 0:7.0.1-1.el7 will be installed
--> Running transaction check---> Package kernel-devel.x86_64 0:3.10.0-1127.18.2.el7 will be installed
---> Package python-bcc.x86_64 0:0.10.0-1.el7 will be installed
---> Package python-netaddr.noarch 0:0.7.5-9.el7 will be installed
--> Finished Dependency ResolutionDependencies Resolved=========================================================================================================================================================================== Package Arch Version Repository Size
===========================================================================================================================================================================Installing: bcc x86_64 0.10.0-1.el7 base 567 k
Installing for dependencies:
bcc-tools x86_64 0.10.0-1.el7 base 394 k
kernel-devel x86_64 3.10.0-1127.18.2.el7 updates 18 M
llvm-private x86_64 7.0.1-1.el7 base 23 M
python-bcc x86_64 0.10.0-1.el7 base 74 k
python-netaddr noarch 0.7.5-9.el7 base 983 k
Transaction Summary===========================================================================================================================================================================Install 1 Package (+5 Dependent packages)
Total download size: 42 M
Installed size: 126 M
Downloading packages:Delta RPMs disabled because /usr/bin/applydeltarpm not installed.(1/6): bcc-tools-0.10.0-1.el7.x86_64.rpm | 394 kB 00:00:05 (2/6): bcc-0.10.0-1.el7.x86_64.rpm | 567 kB 00:00:05 (3/6): python-bcc-0.10.0-1.el7.x86_64.rpm | 74 kB 00:00:00 (4/6): python-netaddr-0.7.5-9.el7.noarch.rpm | 983 kB 00:00:00 (5/6): kernel-devel-3.10.0-1127.18.2.el7.x86_64.rpm | 18 MB 00:00:06 (6/6): llvm-private-7.0.1-1.el7.x86_64.rpm | 23 MB 00:00:30 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------Total 1.4 MB/s | 42 MB 00:00:30 Running transaction checkRunning transaction testTransaction test succeededRunning transaction Installing : llvm-private-7.0.1-1.el7.x86_64 1/6
Installing : kernel-devel-3.10.0-1127.18.2.el7.x86_64 2/6
Installing : python-netaddr-0.7.5-9.el7.noarch 3/6
Installing : python-bcc-0.10.0-1.el7.x86_64 4/6
Installing : bcc-tools-0.10.0-1.el7.x86_64 5/6
Installing : bcc-0.10.0-1.el7.x86_64 6/6
Verifying : python-netaddr-0.7.5-9.el7.noarch 1/6
Verifying : python-bcc-0.10.0-1.el7.x86_64 2/6
Verifying : bcc-0.10.0-1.el7.x86_64 3/6
Verifying : kernel-devel-3.10.0-1127.18.2.el7.x86_64 4/6
Verifying : bcc-tools-0.10.0-1.el7.x86_64 5/6
Verifying : llvm-private-7.0.1-1.el7.x86_64 6/6
Installed: bcc.x86_64 0:0.10.0-1.el7
Dependency Installed: bcc-tools.x86_64 0:0.10.0-1.el7 kernel-devel.x86_64 0:3.10.0-1127.18.2.el7 llvm-private.x86_64 0:7.0.1-1.el7 python-bcc.x86_64 0:0.10.0-1.el7
python-netaddr.noarch 0:0.7.5-9.el7
Complete![root@yang-01 tools]# pwd
/usr/share/bcc/tools[root@yang-01 tools]# ls
argdist cachestat dcstat fileslower javagc mountsnoop opensnoop pythoncalls rubystat sslsniff tcpaccept tplistbashreadline cachetop deadlock filetop javaobjnew mysqld_qslower perlcalls pythonflow runqlat stackcount tcpconnect tracebiolatency capable deadlock.c funccount javastat nfsdist perlflow pythongc runqlen statsnoop tcpconnlat ttysnoopbiosnoop cobjnew doc funclatency javathreads nfsslower perlstat pythonstat runqslower syncsnoop tcpdrop vfscountbiotop cpudist drsnoop funcslower killsnoop nodegc phpcalls reset-trace shmsnoop syscount tcplife vfsstatbitesize cpuunclaimed execsnoop gethostlatency lib nodestat phpflow rubycalls slabratetop tclcalls tcpretrans wakeuptimebpflist dbslower ext4dist hardirqs llcstat offcputime phpstat rubyflow sofdsnoop tclflow tcpsubnet xfsdistbtrfsdist dbstat ext4slower javacalls mdflush offwaketime pidpersec rubygc softirqs tclobjnew tcptop xfsslowerbtrfsslower dcsnoop filelife javaflow memleak oomkill profile rubyobjnew solisten tclstat tcptracer |
点击此处展开...
报错举例1
# 报错举例[root@yang-01 ~]# cachestat 1 3
modprobe: FATAL: Module kheaders not found.chdir(/lib/modules/3.10.0-1127.el7.x86_64/build): No such file or directory
Traceback (most recent call last): File "/usr/share/bcc/tools/cachestat", line 96, in <module>
b = BPF(text=bpf_text)
File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 325, in __init__
raise Exception("Failed to compile BPF text")
Exception: Failed to compile BPF text# 发现报错，这时需要进行内核版本的查看，确保 kernel-devel 与系统上当前安装的内核版本匹配[root@yang-01 tools]# rpm -qa|grep -i kernel | grep -i devel
kernel-devel-3.10.0-1127.18.2.el7.x86_64
[root@yang-01 tools]# uname -r
3.10.0-1127.el7.x86_64
[root@yang-01 tools]# yum install "kernel-devel-uname-r == $(uname -r)"
Loaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirrors.cn99.com
* epel: mirrors.yun-idc.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies--> Running transaction check---> Package kernel-devel.x86_64 0:3.10.0-1127.el7 will be installed
--> Finished Dependency ResolutionDependencies Resolved=========================================================================================================================================================================== Package Arch Version Repository Size
===========================================================================================================================================================================Installing: kernel-devel x86_64 3.10.0-1127.el7 base 18 M
Transaction Summary===========================================================================================================================================================================Install 1 Package
Total download size: 18 M
Installed size: 38 M
Is this ok [y/d/N]: y
Downloading packages:Delta RPMs disabled because /usr/bin/applydeltarpm not installed.kernel-devel-3.10.0-1127.el7.x86_64.rpm | 18 MB 00:00:06 Running transaction checkRunning transaction testTransaction test succeededRunning transaction Installing : kernel-devel-3.10.0-1127.el7.x86_64 1/1
Verifying : kernel-devel-3.10.0-1127.el7.x86_64 1/1
Installed: kernel-devel.x86_64 0:3.10.0-1127.el7
Complete! |
报错举例2
报错: 在安装kernel-devel显示 No package
No package kernel-devel-uname-r == 5.6.2-1.el7.elrepo.x86_64 available.
解决办法: 启用[elrepo-kerne]存储库,# vim /etc/yum.repos.d/elrepo.repo[elrepo-kernel]name=ELRepo.org Community Enterprise Linux Kernel Repository - el7baseurl=http://elrepo.org/linux/kernel/el7/$basearch/
http://mirrors.coreix.net/elrepo/kernel/el7/$basearch/
http://mirror.rackspace.com/elrepo/kernel/el7/$basearch/
http://repos.lax-noc.com/elrepo/kernel/el7/$basearch/
http://mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-kernel.el7
# 修改这一行将0改为1
enabled=1
然后重新执行:# sudo yum install "kernel-devel-uname-r == $(uname -r)"
|
缓存命中查询 命令使用举例
[root@yang-01 tools]# cachestat 1 5
HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
1409 0 77 100.00% 1 2440
1438 0 101 100.00% 1 2440
1413 0 68 100.00% 1 2440
1412 0 117 100.00% 1 2440
1411 0 72 100.00% 1 2440
第一个参数为间隔秒，第二个参数为输出次数TOTAL ，表示总的 I/O 次数MISSES ，表示缓存未命中的次数HITS ，表示缓存命中的次数DIRTIES， 表示新增到缓存中的脏页数BUFFERS_MB 表示 Buffers 的大小，以 MB 为单位CACHED_MB 表示 Cache 的大小，以 MB 为单位 [root@yang-01 tools]# cachetop
16:24:15 Buffers MB: 1 / Cached MB: 2443 / Sort: HITS / Order: ascending
PID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT% 1653 actionte ustats 1 0 0 100.0% 0.0%
1778 actionte umon-mgr 1 0 0 100.0% 0.0%
903 mysql jsvc 1 0 0 100.0% 0.0%
10909 actionte mysqld 2 0 0 100.0% 0.0%
22506 actionte uproxy 2 0 1 50.0% 0.0%
869 mysql jsvc 2 2 2 0.0% 0.0%
10788 actionte mysqld 3 0 0 100.0% 0.0%
1724 actionte urman-agent 4 2 2 33.3% 0.0%
21780 actionte uguard-agent 4 2 2 33.3% 0.0%
12798 actionte ustats 6 0 0 100.0% 0.0%
12799 actionte ustats 6 0 0 100.0% 0.0%
12800 actionte ustats 6 0 0 100.0% 0.0%
这个命令的输出和top类似，默认按照缓存的命中次数（HITS）排序，展示了每个进程的缓存命中情况。这里的 HITS、MISSES 和 DIRTIES ，跟 cachestat 里的含义一样，分别代表间隔时间内的缓存命中次数、未命中次数以及新增到缓存中的脏页数。而 READ_HIT 和 WRITE_HIT ，分别表示读和写的缓存命中率。 |
BCC工具
# 查看新的进程。那些会消耗系统资源，但很短暂的进程，它们甚至不会出现在 top(1) 命令或其它工具中的显示之中。这些新进程可以使用 execsnoop 进行检测（或使用行业术语说，可以被追踪traced）。
[root@yang-01 tools]# execsnoop -h
usage: execsnoop [-h] [-t] [-x] [-q] [-n NAME] [-l LINE] [--max-args MAX_ARGS]Trace exec() syscallsoptional arguments: -h, --help show this help message and exit
-t, --timestamp include timestamp on output
-x, --fails include failed exec()s
-q, --quote Add quotemarks (") around arguments.
-n NAME, --name NAME only print commands matching this name (regex), any
arg
-l LINE, --line LINE only print commands where arg contains this line
(regex)
--max-args MAX_ARGS maximum number of arguments parsed and displayed,
defaults to 20
examples: ./execsnoop # trace all exec() syscalls
./execsnoop -x # include failed exec()s
./execsnoop -t # include timestamps
./execsnoop -q # add "quotemarks" around arguments
./execsnoop -n main # only print command lines containing "main"
./execsnoop -l tpkg # only print command where arguments contains "tpkg"
[root@yang-01 tools]# execsnoop
PCOMM PID PPID RET ARGSbash 17436 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
bash 17437 1642 0
systemctl 17437 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service /usr/bin/systemctl is-active mysqld_3307.service
systemctl 17436 1642 0 /usr/bin/systemctl is-active mysqld_5690.service
bash 17439 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
systemctl 17439 1642 0 /usr/bin/systemctl is-active mysqld_3307.service
bash 17440 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
systemctl 17440 1642 0 /usr/bin/systemctl is-active mysqld_5690.service
bash 17442 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
systemctl 17442 1642 0 /usr/bin/systemctl is-active mysqld_3307.service
bash 17443 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
systemctl 17443 1642 0 /usr/bin/systemctl is-active mysqld_5690.service
bash 17445 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service
bash 17446 1642 0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service
################################################################################################################################################################## # opensnoop 通过跟踪 open() 系统调用来工作。opensnoop 的一些优点在于它能在系统范围内工作，并且跟踪所有进程的 open() 系统调用。[root@yang-01 tools]# opensnoop -h
usage: opensnoop [-h] [-T] [-U] [-x] [-p PID] [-t TID] [-u UID] [-d DURATION] [-n NAME] [-e] [-f FLAG_FILTER]
Trace open() syscallsoptional arguments: -h, --help show this help message and exit
-T, --timestamp include timestamp on output
-U, --print-uid print UID column
-x, --failed only show failed opens
-p PID, --pid PID trace this PID only
-t TID, --tid TID trace this TID only
-u UID, --uid UID trace this UID only
-d DURATION, --duration DURATION
total duration of trace in seconds
-n NAME, --name NAME only print process names containing this name
-e, --extended_fields
show extended fields
-f FLAG_FILTER, --flag_filter FLAG_FILTER
filter on flags argument (e.g., O_WRONLY)
examples: ./opensnoop # trace all open() syscalls
./opensnoop -T # include timestamps
./opensnoop -U # include UID
./opensnoop -x # only show failed opens
./opensnoop -p 181 # only trace PID 181
./opensnoop -t 123 # only trace TID 123
./opensnoop -u 1000 # only trace UID 1000
./opensnoop -d 10 # trace for 10 seconds only
./opensnoop -n main # only print process names containing "main"
./opensnoop -e # show extended fields
./opensnoop -f O_WRONLY -f O_RDWR # only print calls for writing
[root@yang-01 tools]# opensnoop
PID COMM FD ERR PATH1642 ustats 14 0 ./logs
18699 opensnoop -1 2 /usr/lib64/python2.7/encodings/ascii.so
18699 opensnoop -1 2 /usr/lib64/python2.7/encodings/asciimodule.so
18699 opensnoop 16 0 /usr/lib64/python2.7/encodings/ascii.py
18699 opensnoop 17 0 /usr/lib64/python2.7/encodings/ascii.pyc
1309 uagent 16 0 /action-dmp/ustats/ustats.pid
1309 uagent 16 0 /proc/1642/cmdline
1309 uagent 16 0 /action-dmp/umon-mgr/umon-mgr.pid
1309 uagent 16 0 /proc/1684/cmdline
1309 uagent 16 0 /action-dmp/urman-agent/urman-agent.pid
1309 uagent 16 0 /proc/1672/cmdline
1309 uagent 16 0 /action-dmp/ucore/ucore.pid
1309 uagent 16 0 /proc/1538/cmdline
1309 uagent 16 0 /action-dmp/uguard-agent/uguard-agent.pid
1309 uagent 16 0 /proc/1664/cmdline
1309 uagent 16 0 /action-dmp/uguard-mgr/uguard-mgr.pid
1309 uagent 16 0 /proc/1636/cmdline
1309 uagent 16 0 /action-dmp/umon/umon.pid
1309 uagent 16 0 /proc/1734/cmdline
1309 uagent 16 0 /action-dmp/udeploy/udeploy.pid
1309 uagent 16 0 /proc/1644/cmdline
1309 uagent 16 0 /action-dmp/umc/umc.pid
1309 uagent 16 0 /proc/1638/cmdline
1309 uagent 16 0 /action-dmp/uproxy/uproxy.pid
1309 uagent 16 0 /proc/22496/cmdline
1309 uagent 16 0 /action-dmp/urman-mgr/urman-mgr.pid
1309 uagent 16 0 /proc/1640/cmdline
1664 uguard-agent 18 0 ./logs
1642 ustats 14 0 /dev/null
1642 ustats 40 0 /dev/null
18703 bash 3 0 /etc/ld.so.cache
18703 bash 3 0 /lib64/libtinfo.so.5
18703 bash 3 0 /lib64/libdl.so.2
18703 bash 3 0 /lib64/libc.so.6
18703 bash -1 6 /dev/tty
18703 bash 3 0 /usr/lib/locale/locale-archive
18703 bash 3 0 /proc/meminfo
18703 bash 3 0 /usr/lib64/gconv/gconv-modules.cache
18703 bash 3 0 /etc/nsswitch.conf
18703 bash 3 0 /etc/ld.so.cache
18703 bash 3 0 /lib64/libnss_files.so.2
18703 bash 3 0 /etc/passwd
5448 mysqld -1 2 ./universe/u_delay.frm
18703 systemctl 3 0 /etc/ld.so.cache
18703 systemctl 3 0 /lib64/librt.so.1
18703 systemctl 3 0 /lib64/libselinux.so.1
18703 systemctl 3 0 /lib64/liblzma.so.5
18703 systemctl 3 0 /lib64/liblz4.so.1
18703 systemctl 3 0 /lib64/libgcrypt.so.11
18703 systemctl 3 0 /lib64/libcap.so.2
18703 systemctl 3 0 /lib64/libgcc_s.so.1
18703 systemctl 3 0 /lib64/libpthread.so.0
18703 systemctl 3 0 /lib64/libc.so.6
18703 systemctl 3 0 /lib64/libpcre.so.1
18703 systemctl 3 0 /lib64/libdl.so.2
18703 systemctl 3 0 /lib64/libgpg-error.so.0
18703 systemctl 3 0 /lib64/libattr.so.1
18704 bash 3 0 /etc/ld.so.cache
18704 bash 3 0 /lib64/libtinfo.so.5
################################################################################################################################################################## # xfsslower 工具可以跟踪大于 1 毫秒（参数）延迟的常见 XFS 文件系统操作。
# 在 xfsslower 工具运行的时候，通过在 XFS 中动态地检测内核函数实现的，并当它结束的时候解除该检测。# 这个 bcc 工具也有其它文件系统的版本：ext4slower、btrfsslower、zfsslower 和 nfsslower。[root@yang-01 tools]# xfsslower -h
usage: xfsslower [-h] [-j] [-p PID] [min_ms]Trace common XFS file operations slower than a thresholdpositional arguments: min_ms minimum I/O duration to trace, in ms (default 10)
optional arguments: -h, --help show this help message and exit
-j, --csv just print fields: comma-separated values
-p PID, --pid PID trace this PID only
examples: ./xfsslower # trace operations slower than 10 ms (default)
./xfsslower 1 # trace operations slower than 1 ms
./xfsslower -j 1 # ... 1 ms, parsable output (csv)
./xfsslower 0 # trace all operations (warning: verbose)
./xfsslower -p 185 # trace PID 185 only
[root@yang-01 tools]# xfsslower 1
Tracing XFS operations slower than 1 ms
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME17:02:44 umon-mgr 1684 S 0 0 5.93 detail.log
17:02:44 umc 1638 S 0 0 1.05 detail.log
17:02:45 mysqld 10763 W 81920 0 8.12 temp_16.ibt
17:02:45 mysqld 10763 W 16384 80 2.16 temp_16.ibt
17:02:45 ustats 1642 S 0 0 3.12 brief.log
17:02:46 mysqld 10763 W 835584 1024 7.81 ibdata1
17:02:46 mysqld 10763 S 0 0 31.14 ib_logfile0
17:02:46 ucore 1538 S 0 0 5.89 raft.db
17:02:46 ustats 1642 S 0 0 4.04 detail.log
17:02:46 mysqld 10763 S 0 0 3.58 ib_logfile0
17:02:46 ustats 1642 S 0 0 1.49 brief.log
17:02:46 umc 1638 S 0 0 13.70 brief.log
17:02:46 uguard-mgr 1636 S 0 0 9.62 detail.log
17:02:46 ucore 1538 S 0 0 8.87 detail.log
17:02:48 ustats 1642 S 0 0 2.16 brief.log
17:02:49 ustats 1642 S 0 0 1.41 brief.log
17:02:49 uguard-agent 1664 S 0 0 26.30 brief.log
17:02:49 uguard-agent 1664 S 0 0 6.56 detail.log
17:02:49 mysqld 10763 S 0 0 1.89 ib_logfile0
17:02:50 ustats 1642 S 0 0 1.01 brief.log
17:02:50 mysqld 10763 S 0 0 1.18 ib_logfile0
################################################################################################################################################################## # 可以用来实现高效的内核摘要统计。从内核层到用户层的数据传输仅仅是“计数”列。 用户级程序生成其余的。[root@yang-01 tools]# biolatency -h
usage: biolatency [-h] [-T] [-Q] [-m] [-D] [-F] [interval] [count]Summarize block device I/O latency as a histogrampositional arguments: interval output interval, in seconds
count number of outputs
optional arguments: -h, --help show this help message and exit
-T, --timestamp include timestamp on output
-Q, --queued include OS queued time in I/O time
-m, --milliseconds millisecond histogram
-D, --disks print a histogram per disk device
-F, --flags print a histogram per set of I/O flags
examples: ./biolatency # summarize block I/O latency as a histogram
./biolatency 1 10 # print 1 second summaries, 10 times
./biolatency -mT 1 # 1s summaries, milliseconds, and timestamps
./biolatency -Q # include OS queued time in I/O time
./biolatency -D # show each disk device separately
./biolatency -F # show I/O flags separately
[root@yang-01 tools]# biolatency
Tracing block device I/O... Hit Ctrl-C to end.^C usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 13 |* |
32 -> 63 : 306 |********************************* |
64 -> 127 : 367 |****************************************|
128 -> 255 : 254 |*************************** |
256 -> 511 : 12 |* |
512 -> 1023 : 1 | |
1024 -> 2047 : 1 | |
2048 -> 4095 : 9 | |
4096 -> 8191 : 10 |* |
8192 -> 16383 : 9 | |
16384 -> 32767 : 3 | |
32768 -> 65535 : 2 | |
################################################################################################################################################################## # tcplife显示 TCP 会话的生命周期和吞吐量统计[root@yang-01 tools]# tcplife -h
usage: tcplife [-h] [-T] [-t] [-w] [-s] [-p PID] [-L LOCALPORT] [-D REMOTEPORT]
Trace the lifespan of TCP sessions and summarizeoptional arguments: -h, --help show this help message and exit
-T, --time include time column on output (HH:MM:SS)
-t, --timestamp include timestamp on output (seconds)
-w, --wide wide column output (fits IPv6 addresses)
-s, --csv comma separated values output
-p PID, --pid PID trace this PID only
-L LOCALPORT, --localport LOCALPORT
comma-separated list of local ports to trace.
-D REMOTEPORT, --remoteport REMOTEPORT
comma-separated list of remote ports to trace.
examples: ./tcplife # trace all TCP connect()s
./tcplife -t # include time column (HH:MM:SS)
./tcplife -w # wider colums (fit IPv6)
./tcplife -stT # csv output, with times & timestamps
./tcplife -p 181 # only trace PID 181
./tcplife -L 80 # only trace local port 80
./tcplife -L 80,81 # only trace local ports 80 and 81
./tcplife -D 80 # only trace remote port 80
[root@yang-01 tools]# tcplife
PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS854 jsvc ::ffff:127.0.0.1 33222 ::ffff:127.0.0.1 3307 0 0 1.91
10763 mysqld ::ffff:127.0.0.1 3307 ::ffff:127.0.0.1 33222 0 0 1.91
854 jsvc ::ffff:127.0.0.1 33224 ::ffff:127.0.0.1 3307 0 0 2.02
10763 mysqld ::ffff:127.0.0.1 3307 ::ffff:127.0.0.1 33224 0 0 2.03
1638 umc 10.186.60.24 48697 10.186.60.24 5704 0 0 3.01
1636 uguard-mgr ::ffff:10.186.60.24 5704 ::ffff:10.186.60.24 48697 0 0 3.03
854 jsvc ::ffff:127.0.0.1 33228 ::ffff:127.0.0.1 3307 0 0 1.76
10763 mysqld ::ffff:127.0.0.1 3307 ::ffff:127.0.0.1 33228 0 0 1.77
1684 umon-mgr ::1 37504 ::1 8200 0 0 0.03
1684 umon-mgr 127.0.0.1 48056 127.0.0.1 8200 0 0 0.02
854 jsvc ::ffff:127.0.0.1 33234 ::ffff:127.0.0.1 3307 0 0 2.02
10763 mysqld ::ffff:127.0.0.1 3307 ::ffff:127.0.0.1 33234 0 0 2.04
1642 ustats 127.0.0.1 33236 127.0.0.1 3307 0 0 3.13
10763 mysqld ::ffff:127.0.0.1 3307 ::ffff:127.0.0.1 33236 0 0 3.15
1642 ustats 127.0.0.1 45738 127.0.0.1 5690 0 0 3.33
5448 mysqld ::ffff:127.0.0.1 5690 ::ffff:127.0.0.1 45738 0 0 3.35
854 jsvc ::ffff:127.0.0.1 33240 ::ffff:127.0.0.1 3307 0 0 1.65
10763 mysqld ::ffff:127.0.0.1 3307 ::ffff:127.0.0.1 33240 0 0 1.66
#################################################################################################################################################################### # 检测用于名称解析的 gethostbyname(3) 和相关的库调用
[root@yang-01 tools]# gethostlatency -h
usage: gethostlatency [-h] [-p PID]Show latency for getaddrinfo/gethostbyname[2] calls
optional arguments: -h, --help show this help message and exit
-p PID, --pid PID trace this PID only
examples: ./gethostlatency # trace all TCP accept()s
./gethostlatency -p 181 # only trace PID 181
[root@yang-01 tools]# gethostlatency
TIME PID COMM LATms HOST17:37:55 903 jsvc 0.28 localhost
17:38:03 22079 umon 0.28 localhost
17:38:08 14308 umon-mgr 0.23 localhost
17:38:16 919 jsvc 0.17 yang-01
17:38:25 903 jsvc 0.28 localhost
17:38:37 24347 umon 0.25 localhost
17:38:38 1780 umon-mgr 0.34 localhost
17:38:38 22085 umon 0.22 localhost
17:38:43 1780 umon-mgr 0.22 localhost
17:38:45 893 jsvc 0.08 yang-01
##################################################################################################################################################################### # trace 工具由 Sasha Goldshtein 提供，并提供了一些基本的 printf(1) 功能和自定义探针。
[root@yang-01 tools]# trace -h
usage: trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE] [-S] [-M MAX_EVENTS] [-t] [-T] [-C] [-B] [-s SYM_FILE_LIST] [-K]
[-U] [-a] [-I header]
probe [probe ...]
Attach to functions and print trace messages.positional arguments: probe probe specifier (see examples)
optional arguments: -h, --help show this help message and exit
-b BUFFER_PAGES, --buffer-pages BUFFER_PAGES
number of pages to use for perf_events ring buffer
(default: 64)
-p PID, --pid PID id of the process to trace (optional)
-L TID, --tid TID id of the thread to trace (optional)
-v, --verbose print resulting BPF program code before executing
-Z STRING_SIZE, --string-size STRING_SIZE
maximum size to read from strings
-S, --include-self do not filter trace's own pid from the trace
-M MAX_EVENTS, --max-events MAX_EVENTS
number of events to print before quitting
-t, --timestamp print timestamp column (offset from trace start)
-T, --time print time column
-C, --print_cpu print CPU id
-B, --bin_cmp allow to use STRCMP with binary values
-s SYM_FILE_LIST, --sym_file_list SYM_FILE_LIST
coma separated list of symbol files to use for symbol
resolution
-K, --kernel-stack output kernel stack trace
-U, --user-stack output user stack trace
-a, --address print virtual address in stacks
-I header, --include header
additional header files to include in the BPF program
as either full path, or relative to current working
directory, or relative to default kernel header search
path
EXAMPLES:trace do_sys_open Trace the open syscall and print a default trace message when entered
trace 'do_sys_open "%s", arg2'
Trace the open syscall and print the filename being opened
trace 'sys_read (arg3 > 20000) "read %d bytes", arg3'
Trace the read syscall and print a message for reads >20000 bytes
trace 'r::do_sys_open "%llx", retval'
Trace the return from the open syscall and print the return value
trace 'c:open (arg2 == 42) "%s %d", arg1, arg2'
Trace the open() call from libc only if the flags (arg2) argument is 42
trace 'c:malloc "size = %d", arg1'
Trace malloc calls and print the size being allocated
trace 'p:c:write (arg1 == 1) "writing %d bytes to STDOUT", arg3'
Trace the write() call from libc to monitor writes to STDOUT
trace 'r::__kmalloc (retval == 0) "kmalloc failed!"'
Trace returns from __kmalloc which returned a null pointer
trace 'r:c:malloc (retval) "allocated = %x", retval'
Trace returns from malloc and print non-NULL allocated buffers
trace 't:block:block_rq_complete "sectors=%d", args->nr_sector'
Trace the block_rq_complete kernel tracepoint and print # of tx sectors
trace 'u:pthread:pthread_create (arg4 != 0)'
Trace the USDT probe pthread_create when its 4th argument is non-zero
trace 'p::SyS_nanosleep(struct timespec *ts) "sleep for %lld ns", ts->tv_nsec'
Trace the nanosleep syscall and print the sleep duration in ns
trace -I 'linux/fs.h' \
'p::uprobe_register(struct inode *inode) "a_ops = %llx", inode->i_mapping->a_ops'
Trace the uprobe_register inode mapping ops, and the symbol can be found
in /proc/kallsyms
trace -I 'kernel/sched/sched.h' \
'p::__account_cfs_rq_runtime(struct cfs_rq *cfs_rq) "%d", cfs_rq->runtime_remaining'
Trace the cfs scheduling runqueue remaining runtime. The struct cfs_rq is defined
in kernel/sched/sched.h which is in kernel source tree and not in kernel-devel
package. So this command needs to run at the kernel source tree root directory
so that the added header file can be found by the compiler.
trace -I 'net/sock.h' \
'udpv6_sendmsg(struct sock *sk) (sk->sk_dport == 13568)'
Trace udpv6 sendmsg calls only if socket's destination port is equal
to 53 (DNS; 13568 in big endian order)
trace -I 'linux/fs_struct.h' 'mntns_install "users = %d", $task->fs->users'
Trace the number of users accessing the file system of the current task
|
内核版本低的限制
# bcc 基于 eBPF 开发（需要 Linux 3.15 及更高版本）。bcc 使用的大部分内容都需要 Linux 4.1 及更高版本。
[root@yang-01 tools]# dbstat mysql -p 10763 -u
Traceback (most recent call last): File "/usr/share/bcc/tools/dbstat", line 88, in <module>
usdt.enable_probe("query__start", "probe_start")
File "/usr/lib/python2.7/site-packages/bcc/usdt.py", line 154, in enable_probe
probe
bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable
[root@yang-01 tools]# dbslower mysql -p 10763 -m 30
Traceback (most recent call last): File "/usr/share/bcc/tools/dbslower", line 193, in <module>
usdt.enable_probe("query__start", "query_start")
File "/usr/lib/python2.7/site-packages/bcc/usdt.py", line 154, in enable_probe
probe
bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable
# OS内核版本, >= 4.4 存在统计Bug, 部分功能需要>= 4.13
# "bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable” 需要有Dtrace tracepoint的MySQL, 需要重新编译. https://dev.mysql.com/doc/refman/5.7/en/dba-dtrace-server.html
# 使用bcc需要root权限 |
[root@yang-01 tools]# biolatency -D 2
Tracing block device I/O... Hit Ctrl-C to end.
disk = 'vda'
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 63 |*********************************** |
64 -> 127 : 71 |****************************************|
128 -> 255 : 46 |************************* |
256 -> 511 : 4 |** |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 3 |* |
4096 -> 8191 : 3 |* |
disk = ''
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 1 |****************************************|
disk = 'vda'
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 10 |*** |
32 -> 63 : 84 |******************************** |
64 -> 127 : 103 |****************************************|
128 -> 255 : 43 |**************** |
256 -> 511 : 4 |* |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 7 |** |
4096 -> 8191 : 17 |****** |
8192 -> 16383 : 1 | |
16384 -> 32767 : 1 | |
32768 -> 65535 : 1 | |
65536 -> 131071 : 1 | |
131072 -> 262143 : 44 |***************** |
disk = ''
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 1 |****************************************|
[root@yang-01 tools]# ps aux | grep mysql | grep opt
actiont+ 10763 3.3 18.0 4570952 1070628 ? Sl Jul16 1887:06 /opt/mysql/base/8.0.19/bin/mysqld --defaults-file=/opt/mysql/etc/3307/my.cnf --daemonize --pid-file=/opt/mysql/data/3307/mysqld.pid --user=actiontech-mysql --socket=/opt/mysql/data/3307/mysqld.sock --port=3307
[root@yang-01 tools]# filetop -p 10763 -C 5
Tracing... Output every 5 secs. Hit Ctrl-C to end18:01:40 loadavg: 0.14 0.15 0.20 2/1119 31195
TID COMM READS WRITES R_Kb W_Kb T FILE10788 mysqld 0 1 0 832 R ibdata110909 mysqld 19 0 117 0 R mysql-relay.00004410794 mysqld 0 99 0 67 R ib_logfile010977 mysqld 0 60 0 4 R mysql-relay.00004410910 mysqld 0 10 0 4 R mysql-bin.00001510789 mysqld 0 1 0 0 R ib_logfile018:01:45 loadavg: 0.21 0.16 0.21 2/1119 31212
TID COMM READS WRITES R_Kb W_Kb T FILE10909 mysqld 19 0 116 0 R mysql-relay.00004410794 mysqld 0 98 0 64 R ib_logfile010977 mysqld 0 60 0 4 R mysql-relay.00004410910 mysqld 0 10 0 4 R mysql-bin.00001518:01:50 loadavg: 0.19 0.16 0.20 2/1119 31226
TID COMM READS WRITES R_Kb W_Kb T FILE10909 mysqld 15 0 88 0 R mysql-relay.00004410794 mysqld 0 106 0 70 R ib_logfile019780 mysqld 2 2 8 7 R ibzMa2hy10977 mysqld 0 60 0 4 R mysql-relay.00004410910 mysqld 0 10 0 4 R mysql-bin.00001519780 mysqld 1 0 0 0 R mysql-bin.00000919780 mysqld 1 0 0 0 R mysql-bin.00001219780 mysqld 1 0 0 0 R mysql-bin.00001419780 mysqld 1 0 0 0 R mysql-bin.00001019780 mysqld 1 0 0 0 R mysql-bin.00001319780 mysqld 1 0 0 0 R mysql-bin.000011[root@yang-01 tools]# filelife
TIME PID COMM AGE(s) FILE18:03:26 1823 ustats 0.00 disk-read-write-test.log
18:03:26 1830 ustats 0.00 disk-read-write-test.log
18:03:29 1736 umon-mgr 0.01 grafana.db-journal18:03:41 1823 ustats 0.00 disk-read-write-test.log
18:03:41 1784 ustats 0.01 disk-read-write-test.log
|