KVM虚拟机内无agent情况下的监控（ceilometer实现）
今天看到大家在群里讨论KVM虚拟机的监控问题，而且是要求VM内无agent情况下的监控。这方面确实没有深入研究，但尚有些openstack/ceilometer的使用经验，略改过些源码。应肖力大哥号召，十分愿意与大家分享经验，错误与不足之处还望大家雅正。
ceilometer介绍
这里http://docs.openstack.org/developer/ceilometer/architecture.html对ceilometer有些大致的介绍，简单说就是OpenStack中的监控（遥测）项目，完成对云计算环境下各个指标的监控，事件记录，报警预警等。
监控方面大致包括数据中心的温度，宿主机CPU温度，电压，电流，风扇运转情况，网络负载情况，CPU使用情况；虚拟机CPU使用率，内存使用情况，每张网卡的I/O情况，每块磁盘的I/O情况等。
当然按ceilometer的设计来说也支持OpenStack以外的环境使用，并且它提供了比较优秀的软件架构，只需完成小部分代码就可以方便的添加想要监控的指标，比如物理节点磁盘健康状态。
然而现在的问题是：随着时间的推移和云系统规模的扩大ceilometer获取的数据量比较大，在使用上出现查询延迟现象比较严重，接着出现了gnocchi这个项目来解决，但是问题不少，之前一段时间重点就是优化这个问题。
ceilometer监控虚拟机原理
ceilometer中有个组件叫ceilometer-agent-compute运行在计算节点（即VM宿主机）上，来完成对VM各项指标监控，支持HyperV，Libvirt，VMWare，XEN。
KVM/QEMU的监控代码详见：https://github.com/openstack/ceilometer/blob/master/ceilometer/compute/virt/libvirt/inspector.py
总体思路是ceilometer-agent-compute定时轮查本计算节点上的每个虚拟机，使用libvirt的API获取domain的情况，根据xml配置文件获取每个可用的要监测设备的名称，再查看domain设备的属性字段值即是要监控的数值。
获取宿主机上每个instance的domain
获取宿主机上每个instance的domain，其属性如下，可以看到很多有用的信息。
具体的参数值代表的意思请参考http://libvirt.org/html/libvirt-libvirt-domain.html对应的API。
比如下文用到的interfaceStats：
http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainInterfaceStats
virDomainInterfaceStats
http://libvirt.org/html/libvirt-libvirt-domain.html#virDomainInterfaceStatsStruct
struct virDomainInterfaceStatsStruct {
long long
rx_bytes
long long
rx_packets
long long
rx_errs
long long
rx_drop
long long
tx_bytes
long long
tx_packets
long long
tx_errs
long long
tx_drop
}
(Pdb) dir(domain)
['ID','OSType', 'UUID', 'UUIDString', 'XMLDesc', '__class__', '__del__','__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__','__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__','__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__','__weakref__', '_conn', '_o', 'abortJob', 'attachDevice', 'attachDeviceFlags','autostart', 'blkioParameters', 'blockCommit', 'blockCopy', 'blockInfo','blockIoTune', 'blockJobAbort', 'blockJobInfo', 'blockJobSetSpeed','blockPeek', 'blockPull', 'blockRebase', 'blockResize', 'blockStats', 'blockStatsFlags','connect', 'controlInfo', 'coreDump', 'coreDumpWithFormat', 'create','createWithFiles', 'createWithFlags', 'destroy', 'destroyFlags','detachDevice', 'detachDeviceFlags', 'diskErrors', 'emulatorPinInfo', 'fSTrim','fsFreeze', 'fsThaw', 'getCPUStats', 'getTime', 'hasCurrentSnapshot','hasManagedSaveImage', 'hostname', 'info', 'injectNMI', 'interfaceParameters', 'interfaceStats','isActive', 'isPersistent', 'isupdated', 'jobInfo', 'jobStats', 'listAllSnapshots','managedSave', 'managedSaveRemove', 'maxMemory', 'maxVcpus','memoryParameters', 'memoryPeek', 'memoryStats', 'metadata', 'migrate', 'migrate2','migrate3', 'migrateGetCompressionCache', 'migrateGetMaxSpeed','migrateSetCompressionCache', 'migrateSetMaxDowntime', 'migrateSetMaxSpeed','migrateToURI', 'migrateToURI2', 'migrateToURI3', 'name', 'numaParameters','openChannel', 'openConsole', 'openGraphics', 'openGraphicsFD','pMSuspendForDuration', 'pMWakeup', 'pinEmulator', 'pinVcpu', 'pinVcpuFlags','reboot', 'reset', 'resume', 'revertToSnapshot', 'save', 'saveFlags','schedulerParameters', 'schedulerParametersFlags', 'schedulerType','screenshot', 'securityLabel', 'securityLabelList', 'sendKey','sendProcessSignal', 'setAutostart', 'setBlkioParameters', 'setBlockIoTune','setInterfaceParameters', 'setMaxMemory', 'setMemory', 'setMemoryFlags','setMemoryParameters', 'setMemoryStatsPeriod', 'setMetadata','setNumaParameters', 'setSchedulerParameters', 'setSchedulerParametersFlags','setTime', 'setVcpus', 'setVcpusFlags', 'shutdown', 'shutdownFlags','snapshotCreateXML', 'snapshotCurrent', 'snapshotListNames','snapshotLookupByName', 'snapshotNum', 'state', 'suspend', 'undefine','undefineFlags', 'updateDeviceFlags', 'vcpuPinInfo', 'vcpus', 'vcpusFlags']
监控CPU
# number: number of CPUs
# time: cumulative CPU time
(Pdb) p dom_info
[1, 4194304L,4194304L, 4, 89760510000000L]
def inspect_cpus(self, instance):
domain = self._lookup_by_uuid(instance)
dom_info= domain.info()
return virt_inspector.CPUStats(number=dom_info[3],
time=dom_info[4])
监控网卡
# name: the name of the vNIC
# mac: the MAC address
# fref: the filter ref
# parameters: miscellaneous parameters
# rx_bytes: number of received bytes
# rx_packets: number of received packets
# tx_bytes: number of transmitted bytes
# tx_packets: number of transmitted packets
# rx_bytes_rate: rate of received bytes
# tx_bytes_rate: rate of transmitted bytes
(Pdb) dir(tree)
['__class__', '__contains__', '__copy__','__deepcopy__', '__delattr__', '__delitem__', '__doc__', '__format__','__getattribute__', '__getitem__', '__hash__', '__init__', '__iter__','__len__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__','__reversed__', '__setattr__', '__setitem__', '__sizeof__', '__str__','__subclasshook__', '_init', 'addnext', 'addprevious', 'append', 'attrib','base', 'clear', 'extend', 'find', 'findall', 'findtext', 'get', 'getchildren','getiterator', 'getnext', 'getparent', 'getprevious', 'getroottree', 'index','insert', 'items', 'iter', 'iterancestors', 'iterchildren', 'iterdescendants','iterfind', 'itersiblings', 'itertext', 'keys', 'makeelement', 'nsmap','prefix', 'remove', 'replace', 'set', 'sourceline', 'tag', 'tail', 'text','values', 'xpath']
根据xml的tree可以获得instance各类每个设备的名称来调用libvirt的API获得相应的信息。
(Pdb) p interface
Interface(name='tape7efdecc-4c',mac='fa:16:3e:59:cd:bc', fref=None, parameters={})
(Pdb) p dom_stats
(788489893L,1004383L, 0L, 0L, 507858406L, 748909L, 0L, 0L)
(Pdb) p stats
InterfaceStats(rx_bytes=788489893L,rx_packets=1004383L, tx_bytes=507858406L, tx_packets=748909L)
def inspect_vnics(self, instance):
domain = self._get_domain_not_shut_off_or_raise(instance)
tree = etree.fromstring(domain.XMLDesc(0))
for iface in tree.findall('devices/interface'):
target = iface.find('target')
if target is not None:
name = target.get('dev')
else:
continue
mac = iface.find('mac')
if mac is not None:
mac_address =mac.get('address')
else:
continue
fref = iface.find('filterref')
if fref is not None:
fref = fref.get('filter')
params = dict((p.get('name').lower(), p.get('value'))
for p in iface.findall('filterref/parameter'))
interface = virt_inspector.Interface(name=name,
mac=mac_address,
fref=fref,parameters=params)
dom_stats= domain.interfaceStats(name)
stats =virt_inspector.InterfaceStats(rx_bytes=dom_stats[0],
rx_packets=dom_stats[1],
tx_bytes=dom_stats[4],
tx_packets=dom_stats[5])
yield (interface, stats)
监控磁盘信息
# capacity: capacity of the disk
# allocation: allocation of the disk
# physical: usage of the disk
(Pdb)p block_info
[85899345920L, 1009991680L,1009991680L]
(Pdb) p info
DiskInfo(capacity=85899345920L,allocation=1009991680L,physical=1009991680L)
def inspect_disk_info(self, instance):
domain = self._get_domain_not_shut_off_or_raise(instance)
tree = etree.fromstring(domain.XMLDesc(0))
for device in filter(
bool,
[target.get("dev")
for target in tree.findall('devices/disk/target')]):
disk = virt_inspector.Disk(device=device)
block_info= domain.blockInfo(device)
info =virt_inspector.DiskInfo(capacity=block_info[0],
allocation=block_info[1],
physical=block_info[2])
监控磁盘I/O情况
# disk_latency: average disk latency
# iops: number of iops per second
# read_bytes: number of bytes read
# read_requests: number of read operations
# write_bytes: number of bytes written
# write_requests: number of writeoperations
# errors: number of errors
# read_bytes_rate: number of bytes read persecond
# read_requests_rate: number of readoperations per second
# write_bytes_rate: number of bytes writtenper second
# write_requests_rate: number of writeoperations per second
(Pdb) p block_stats
(67017L, 1412453376L,245180L, 2315730432L, -1L)
(Pdb) p stats
DiskStats(read_bytes=1412453376L,read_requests=67017L, write_bytes=2315730432L, write_requests=245180L,errors=-1L)
def inspect_disks(self, instance):
domain = self._get_domain_not_shut_off_or_raise(instance)
tree = etree.fromstring(domain.XMLDesc(0))
for device in filter(
bool,
[target.get("dev")
for target in tree.findall('devices/disk/target')]):
disk = virt_inspector.Disk(device=device)
block_stats= domain.blockStats(device)
stats =virt_inspector.DiskStats(read_requests=block_stats[0],
read_bytes=block_stats[1],
write_requests=block_stats[2],
write_bytes=block_stats[3],
errors=block_stats[4])
yield (disk, stats)
监控内存使用情况
参考：
http://blog.sina.com.cn/s/blog_6de3aa8a0102vgoo.html
http://paste.openstack.org/show/78624/
https://fedoraproject.org/wiki/Windows_Virtio_Drivers
http://www.linux-kvm.org/page/Virtio
# util: CPU utilization in percentage
# usage: Amount of memory used
# resident: Amount of resident memory
virsh dommemstat domain-id
actual 4194304
swap_in 0
swap_out 0
major_fault 0
minor_fault 8
unused 3572748
available 4193904
rss 4359584
要获取VM内存使用详细信息，VM中需要安装virtio驱动并且支持memballoon。
To enable the libvirt memory.usage supporting, you need libvirt version1.1.1+, qemu version 1.5+,and you need to prepare suitable balloon driver in the image, particularly for Windows guests, most modern Linuxes have it built in. The memory.usage meters can't be fetched without image balloon driver.
关于virtio驱动：Linux的一般都会包含( lsmod | grep virtio 查看)，但是windows的virtio驱动需要自己在镜像中安装。
Virtio for windows下载地址：
https://launchpad.net/kvm-guest-drivers-windows/+download
注意系统的版本与驱动的版本对应：
步骤：将驱动文件夹中的BLNSVR.EXE拷贝进系统目录C:Windowssystem32，然后使用”BLNSVR -I”命令安装服务。
(Pdb) p memory_stats
{'swap_out': 0L,'available': 4193904L, 'actual': 4194304L, 'major_fault': 0L, 'swap_in': 0L,'unused': 3572748L, 'minor_fault': 8L, 'rss': 4359600L}
(Pdb) p memory_used
606L
def inspect_memory_usage(self, instance, duration=None):
instance_name = util.instance_name(instance)
domain = self._get_domain_not_shut_off_or_raise(instance)
try:
memory_stats= domain.memoryStats()
if (memory_stats and
memory_stats.get('available')and
memory_stats.get('unused')):
memory_used= (memory_stats.get('available') -
memory_stats.get('unused'))
# Stat provided from libvirt isin KB, converting it to MB.
memory_used = memory_used /units.Ki
returnvirt_inspector.MemoryUsageStats(usage=memory_used)
else:
............
............
(Pdb) p memory
4257L
def inspect_memory_resident(self, instance, duration=None):
domain = self._get_domain_not_shut_off_or_raise(instance)
memory =domain.memoryStats()['rss'] / units.Ki
return virt_inspector.MemoryResidentStats(resident=memory)
自己写监控时可以参考ceilometer的实现代码。以上代码获取的都是基础数据，再进行数据加工处理即可获得想要的数据了（比如CPU使用率，内存使用率，磁盘IOPS，网络吞吐等）。
关于VM监控数值的准确程度，确实没有细致的测试过，但是差别肯定是有的，就看能否接受了。即使不准确也还是可以根据获取数据的原理和数据处理的方法来逼近实际值，达到一个能够接受的误差范围之内。其实100%的准确恐怕也没这个必要，大多数情况下监控数据还是作为统计量来使用，做预警告警时也可以根据实际误差来灵活调整阈值。
谢谢大家，欢迎雅正。
作者姓名：张国庆
邮箱：zhangguoqingas@gmail.com
博客：http://blog.sina.com.cn/aszhangguoqing
附-实测情况
Windows server 2008-R2: vCPU 4 RAM 4GB DISK 80GB single-Network-interface
