openstack 云主机关机：ournal: End of file while reading data: Input/output e...
问题现象描述： 某个集群环境每天出现一台机器关机现象，随机发生，经过排查解决问题，为大家提供方便 环境： 集群环境：openstack ＋ ceph 融合集群，版本：Mitaka＋jewel 网络环境：网卡10G＋bond0(主备模式) 版 本：centos7.3 message 错误日志： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Aug3016:45:14lxx-4-5journal:internalerror:Endof file frommonitor Aug3016:45:14lxx-4-5avahi-daemon[2412]:Withdrawingaddressrecord for fe80::fc16:3eff:fef3:5076onvnet5. Aug3016:45:14lxx-4-5kernel:vlan206:port3(vnet5)entereddisabledstate Aug3016:45:14lxx-4-5kvm:10guestsnowactive Aug3016:45:14lxx-4-5avahi-daemon[2412]:Withdrawingworkstationservice for vnet5. Aug3016:45:14lxx-4-5kernel:devicevnet5leftpromiscuousmode Aug3016:45:14lxx-4-5kernel:vlan206:port3(vnet5)entereddisabledstate Aug3016:45:14lxx-4-5systemd:autolog.serviceholdoff time over,schedulingrestart. Aug3016:45:14lxx-4-5systemd:StartedAutolog. Aug3016:45:14lxx-4-5systemd:StartingAutolog... Aug3016:45:14lxx-4-5systemd-machined:Machineqemu-22-instance-000002c9terminated. Aug3016:45:14lxx-4-5autolog:Don'thavemasterprocess. Aug3016:45:15l22-4-5journal:Endof file while readingdata:Input /output error Aug3016:45:15lxx-4-5systemd:autolog.serviceholdoff time over,schedulingrestart. Aug3016:45:15lxx-4-5systemd:StartedAutolog. Aug3016:45:15lxx-4-5systemd:StartingAutolog... Aug3016:45:15lxx-4-5autolog:Don'thavemasterprocess. Aug3016:45:15lxx-4-5systemd:autolog.serviceholdoff time over,schedulingrestart. Aug3016:45:15lxx-4-5systemd:StartedAutolog. Aug3016:45:15lxx-4-5systemd:StartingAutolog... openstack-compute 关键日志： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2017 - 08 - 30 16 : 45 : 20.952 110867 DEBUGnova.compute.manager[req-0602316d-944c-42b4-9d3c-7d1b0e513765-----][instance:26f48b2e-f648-42e2- 8133 -7ebc060fd7ae]Updatedthenetworkinfo_cache for instance_heal_instance_info_cache/usr/lib/python2. 7 /site-packages/nova/compute/manager.py: 5803 2017 - 08 - 30 16 : 45 : 30.033 110867 DEBUGnova.virt.driver[-]Emittingevent<LifecycleEvent: 1504082715.03 ,08330b10-f106- 4737 -b9db-0e45c84abb2e=>Stopped>emit_event/usr/lib/python2. 7 /site-packages/nova/virt/driver.py: 1443 2017 - 08 - 30 16 : 45 : 30.034 110867 INFOnova.compute.manager[-][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]VMStopped(LifecycleEvent) 2017 - 08 - 30 16 : 45 : 30.076 110867 DEBUGnova.compute.manager[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Checkingstate_get_power_state/usr/lib/python2. 7 /site-packages/nova/compute/manager.py: 1347 2017 - 08 - 30 16 : 45 : 30.079 110867 DEBUGnova.compute.manager[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Synchronizinginstancepowerstateafterlifecycleevent "Stopped" ;currentvm_state:active,currenttask_state:None,currentDBpower_state: 1 ,VMpower_state: 4 handle_lifecycle_event/usr/lib/python2. 7 /site-packages/nova/compute/manager.py: 1276 2017 - 08 - 30 16 : 45 : 30.119 110867 INFOnova.compute.manager[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]During_sync_instance_power_statetheDBpower_state( 1 )doesnotmatchthevm_power_statefromthehypervisor( 4 ).Updatingpower_state in theDBtomatchthehypervisor. 2017 - 08 - 30 16 : 45 : 30.177 110867 WARNINGnova.compute.manager[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Instanceshutdownbyitself.CallingthestopAPI.Currentvm_state:active,currenttask_state:None,originalDBpower_state: 1 ,currentVMpower_state: 4 2017 - 08 - 30 16 : 45 : 30.178 110867 DEBUGnova.compute.api[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Goingto try tostopinstanceforce_stop/usr/lib/python2. 7 /site-packages/nova/compute/api.py: 1954 2017 - 08 - 30 16 : 45 : 30.267 110867 DEBUGoslo_concurrency.lockutils[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----]Lock "08330b10-f106-4737-b9db-0e45c84abb2e" acquiredby "nova.compute.manager.do_stop_instance" ::waited 0 .000sinner/usr/lib/python2. 7 /site-packages/oslo_concurrency/lockutils.py: 270 2017 - 08 - 30 16 : 45 : 30.268 110867 DEBUGnova.compute.manager[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Checkingstate_get_power_state/usr/lib/python2. 7 /site-packages/nova/compute/manager.py: 1347 2017 - 08 - 30 16 : 45 : 30.270 110867 DEBUGnova.compute.manager[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Stoppinginstance;currentvm_state:active,currenttask_state:powering-off,currentDBpower_state: 4 ,currentVMpower_state: 4 do_stop_instance/usr/lib/python2. 7 /site-packages/nova/compute/manager.py: 2545 2017 - 08 - 30 16 : 45 : 30.270 110867 INFOnova.compute.manager[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Instance is alreadypoweredoff in thehypervisorwhenstop is called. 2017 - 08 - 30 16 : 45 : 30.271 110867 DEBUGnova.objects.instance[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----]Lazy-loading 'metadata' onInstanceuuid08330b10-f106- 4737 -b9db-0e45c84abb2eobj_load_attr/usr/lib/python2. 7 /site-packages/nova/objects/instance.py: 895 2017 - 08 - 30 16 : 45 : 30.314 110867 INFOnova.virt.libvirt.driver[req-5998b542-495c-41f2- 8010 -7f1c426f0127-----][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Instancealreadyshutdown. 2017 - 08 - 30 16 : 45 : 30.318 110867 INFOnova.virt.libvirt.driver[-][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]Instancedestroyedsuccessfully. 关键日志： 1 2 3 message:Aug 30 16 : 45 : 15 l22- 4 - 5 journal:Endoffile while readingdata:Input/outputerror Openstack-compute: 2017 - 08 - 30 16 : 45 : 30.034 110867 INFOnova.compute.manager[-][instance:08330b10-f106- 4737 -b9db-0e45c84abb2e]VMStopped(LifecycleEvent) 解决办法： 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 升级libvirt版本： libvirt-daemon-driver-secret- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-lxc- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-driver-lxc- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-python- 2.0 . 0 - 2 .el7.x86_64 libvirt-daemon- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-lock-sanlock- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-driver-storage- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-gobject- 0.2 . 3 - 1 .el7.x86_64 libvirt-nss- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-driver-nwfilter- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-gconfig- 0.2 . 3 - 1 .el7.x86_64 libvirt-snmp- 0.0 . 3 - 5 .el7.x86_64 libvirt-daemon-driver-nodedev- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-glib-devel- 0.2 . 3 - 1 .el7.x86_64 libvirt-gobject-devel- 0.2 . 3 - 1 .el7.x86_64 libvirt-java-javadoc- 0.4 . 9 - 4 .el7.noarch libvirt-daemon-driver-qemu- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-kvm- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-gconfig-devel- 0.2 . 3 - 1 .el7.x86_64 libvirt-login-shell- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-client- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-driver- interface - 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-devel- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-cim- 0.6 . 3 - 19 .el7.x86_64 libvirt-glib- 0.2 . 3 - 1 .el7.x86_64 libvirt-java-devel- 0.4 . 9 - 4 .el7.noarch libvirt-daemon-driver-network- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-docs- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-config-nwfilter- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-daemon-config-network- 2.0 . 0 - 10 .el7_3. 9 .x86_64 libvirt-java- 0.4 . 9 - 4 .el7.noarch 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 升级qemu版本 qemu-system-lm32- 2.0 . 0 - 1 .el7. 6 .x86_64 ipxe-roms-qemu- 20160127 - 5 .git6366fa7a.el7.noarch qemu-system-cris- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-x86- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-kvm-tools- 1.5 . 3 - 126 .el7_3. 10 .x86_64 qemu-system-xtensa- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-arm- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-s390x- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-sh4- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-kvm-common- 1.5 . 3 - 126 .el7_3. 10 .x86_64 qemu-user- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-unicore32- 2.0 . 0 - 1 .el7. 6 .x86_64 libvirt-daemon-driver-qemu- 2.0 . 0 - 10 .el7_3. 9 .x86_64 qemu-guest-agent- 2.5 . 0 - 3 .el7.x86_64 qemu-common- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-or32- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-kvm- 1.5 . 3 - 126 .el7_3. 10 .x86_64 qemu-system-moxie- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-img- 1.5 . 3 - 126 .el7_3. 10 .x86_64 qemu-system-m68k- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-alpha- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-microblaze- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu-system-mips- 2.0 . 0 - 1 .el7. 6 .x86_64 qemu- 2.0 . 0 - 1 .el7. 6 .x86_64 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 升级kernel [root@~]#rpm-qa|grepkernel kernel- 3.10 . 0 - 514.26 . 2 .el7.x86_64 kernel-tools-libs- 3.10 . 0 - 514.26 . 2 .el7.x86_64 kernel-devel- 3.10 . 0 - 327.36 . 3 .el7.x86_64 kernel-tools- 3.10 . 0 - 514.26 . 2 .el7.x86_64 kernel-devel- 3.10 . 0 - 123 .el7.x86_64 kernel- 3.10 . 0 - 327.36 . 3 .el7.x86_64 abrt-addon-kerneloops- 2.1 . 11 - 45 .el7.centos.x86_64 kernel- 3.10 . 0 - 514.2 . 2 .el7.x86_64 kernel- 3.10 . 0 - 123 .el7.x86_64 kernel- 3.10 . 0 - 327.22 . 2 .el7.x86_64 kernel-devel- 3.10 . 0 - 327.22 . 2 .el7.x86_64 kernel-devel- 3.10 . 0 - 514.26 . 2 .el7.x86_64 kernel-devel- 3.10 . 0 - 514.2 . 2 .el7.x86_64 kernel-headers- 3.10 . 0 - 514.26 . 2 .el7.x86_64 [root@~]#uname-r 3.10 . 0 - 514.26 . 2 .el7.x86_64 注意：升级版本之后一定要重启，才能成功，重启服务无效！！！ 本文转自 swq499809608 51CTO博客，原文链接:http://blog.51cto.com/swq499809608/1962081
