> On 27 Jun 2018, at 16:47, Michael Watters <[email protected]> wrote: > > Having the same issue after upgrading from ovirt 4.1 to 4.2.4. > Attempting to migrate any VM results in an immediate failure. The > engine log shows an error as follows. > > 2018-06-27 10:43:59,815-04 ERROR > [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] > (ForkJoinPool-1-worker-3) [] Migration of VM 'ASAP' to host > 'ovirt-node-production1' failed: VM destroyed during the startup. > 2018-06-27 10:43:59,819-04 INFO > [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] > (ForkJoinPool-1-worker-10) [] VM > '91be7864-3793-4583-accb-31b441af7b6a'(ASAP) moved from 'MigratingFrom' > --> ‘Up'
please include vdsm.logs from source and destination hypervisors Thanks, michal > > > On 06/19/2018 04:29 AM, [email protected] wrote: >> Hello, >> >> I'm using a 4 node cluster oVirt 4.1. After updating the Engine to 4.2.3.8 >> (and ContOS 7.5), I tried to update the nodes too. >> Unfortunally, the live migration is not working (any more). Maybe it's >> related to following error found in engine.log >> >> 2018-06-19 09:32:53,469+02 WARN >> [org.ovirt.engine.core.bll.provider.network.openstack.CustomizedRESTEasyConnector] >> (EE-ManagedThreadFactory-engineScheduled-Thread-7) [4d6a7dc4] Cannot >> register external providers trust store: java.io.IOException: Keystore was >> tampered with, or password was incorrect >> 2018-06-19 09:32:53,475+02 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-7) [4d6a7dc4] Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error unable to find valid >> certification path to requested target and code 5050) >> >> I'm using the selfsigned CA provided with oVirt. >> >> Nodes (source and destination) >> OS-Version:RHEL - 7 - 4.1708.el7.centos >> Kernelversion: 3.10.0 - 693.21.1.el7.x86_64 >> KVM-Version:2.9.0 - 16.el7_4.14.1 >> LIBVIRT-Version:libvirt-3.2.0-14.el7_4.9 >> VDSM-Version:vdsm-4.19.45-1.el7.centos >> SPICE-Version:0.12.8 - 2.el7.1 >> CEPH-Version:librbd1-0.94.5-2.el7 >> Kernel Features:PTI: 1, IBPB: 0, IBRS: 0 >> >> engine.log for the mirgration: >> 2018-06-14 13:13:30,300+02 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-246968) [] EVENT_ID: >> VM_MIGRATION_START_SYSTEM_INITIATED(67), Migration initiated by system (VM: >> vm_to_migrate, Source: SOURCE, Destination: DESTINATION, Reason: Host >> preparing for maintenance). >> 2018-06-14 13:13:32,886+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] >> (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] Fetched 8 VMs from >> VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498' >> 2018-06-14 13:13:32,887+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:13:32,887+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:13:47,939+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-42) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:13:47,939+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-42) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:14:02,993+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:14:02,993+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:14:18,042+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:14:18,042+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:14:33,090+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:14:33,090+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:14:38,356+02 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Lock Acquired >> to object >> 'EngineLock:{exclusiveLocks='[d1457e7e-f29d-49ac-b635-2457a7cbc59f=PROVIDER]', >> sharedLocks=''}' >> 2018-06-14 13:14:38,369+02 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Running >> command: SyncNetworkProviderCommand internal: true. >> 2018-06-14 13:14:38,370+02 WARN >> [org.ovirt.engine.core.bll.provider.network.openstack.CustomizedRESTEasyConnector] >> (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Cannot >> register external providers trust store: java.io.IOException: Keystore was >> tampered with, or password was incorrect >> 2018-06-14 13:14:38,379+02 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error unable to find valid >> certification path to requested target and code 5050) >> 2018-06-14 13:14:38,382+02 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Lock freed to >> object >> 'EngineLock:{exclusiveLocks='[d1457e7e-f29d-49ac-b635-2457a7cbc59f=PROVIDER]', >> sharedLocks=''}' >> 2018-06-14 13:14:48,139+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-92) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:14:48,139+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-92) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:15:03,193+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:15:03,193+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:15:18,242+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:15:18,242+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:15:33,293+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly >> detected as 'MigratingTo' on VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on >> 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a') >> 2018-06-14 13:15:33,293+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS >> 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the >> refresh until migration is done >> 2018-06-14 13:15:45,377+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-10) [] VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) moved from >> 'MigratingFrom' --> 'Up' >> 2018-06-14 13:15:45,377+02 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (EE-ManagedThreadFactory-engineScheduled-Thread-10) [] Adding VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) to re-run list >> 2018-06-14 13:15:45,400+02 ERROR >> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] >> (EE-ManagedThreadFactory-engineScheduled-Thread-10) [] Rerun VM >> '4aff4193-ba75-481d-92b3-59b62cd8b111'. Called from VDS 'SOURCE' >> 2018-06-14 13:15:45,451+02 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] START, >> MigrateStatusVDSCommand(HostName = SOURCE, >> MigrateStatusVDSCommandParameters:{hostId='ea7bae8f-d966-4baa-b8b4-a5522b88ec3a', >> vmId='4aff4193-ba75-481d-92b3-59b62cd8b111'}), log id: 25f95334 >> 2018-06-14 13:15:45,453+02 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] FINISH, >> MigrateStatusVDSCommand, log id: 25f95334 >> 2018-06-14 13:15:45,478+02 WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] EVENT_ID: >> VM_MIGRATION_TRYING_RERUN(128), Failed to migrate VM vm_to_migrate to Host >> DESTINATION . Trying to migrate to another Host. >> 2018-06-14 13:15:45,557+02 INFO >> [org.ovirt.engine.core.bll.MigrateVmCommand] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] Running command: >> MigrateVmCommand internal: true. Entities affected : ID: >> 4aff4193-ba75-481d-92b3-59b62cd8b111 Type: VMAction group MIGRATE_VM with >> role type USER >> 2018-06-14 13:15:45,595+02 INFO >> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] START, MigrateVDSCommand( >> MigrateVDSCommandParameters:{hostId='ea7bae8f-d966-4baa-b8b4-a5522b88ec3a', >> vmId='4aff4193-ba75-481d-92b3-59b62cd8b111', srcHost='10.157.8.37', >> dstVdsId='69ebacd7-ee55-4c52-abf1-437a14d5fb0d', >> dstHost='10.157.8.42:54321', migrationMethod='ONLINE', >> tunnelMigration='false', migrationDowntime='0', autoConverge='false', >> migrateCompressed='false', consoleAddress='null', maxBandwidth='500', >> enableGuestEvents='false', maxIncomingMigrations='2', >> maxOutgoingMigrations='2', convergenceSchedule='null', >> dstQemu='192.168.1.113'}), log id: 736d023f >> 2018-06-14 13:15:45,595+02 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] START, >> MigrateBrokerVDSCommand(HostName = SOURCE, >> MigrateVDSCommandParameters:{hostId='ea7bae8f-d966-4baa-b8b4-a5522b88ec3a', >> vmId='4aff4193-ba75-481d-92b3-59b62cd8b111', srcHost='10.157.8.37', >> dstVdsId='69ebacd7-ee55-4c52-abf1-437a14d5fb0d', >> dstHost='10.157.8.42:54321', migrationMethod='ONLINE', >> tunnelMigration='false', migrationDowntime='0', autoConverge='false', >> migrateCompressed='false', consoleAddress='null', maxBandwidth='500', >> enableGuestEvents='false', maxIncomingMigrations='2', >> maxOutgoingMigrations='2', convergenceSchedule='null', >> dstQemu='192.168.1.113'}), log id: 58572c5b >> 2018-06-14 13:15:45,601+02 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] FINISH, >> MigrateBrokerVDSCommand, log id: 58572c5b >> 2018-06-14 13:15:45,604+02 INFO >> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-247023) [] FINISH, MigrateVDSCommand, >> return: MigratingFrom, log id: 736d023f >> >> _______________________________________________ >> Users mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/[email protected]/message/B6IEVX5XEBFHJCWCHGRMPOJ27VUJI2TL/ > _______________________________________________ > Users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/[email protected]/message/63VCY4HBRVBVIBQVCIWMQPANEKTXMBG4/ _______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/WRV7MMQMW27HFWNM45RJPZJBHEHWCFRM/
