Hi all,
I have a problem with failing vmotion or storage vmotion for some VMs in an vSphere 5.1 (build 1065491). In both cases I get the following error: "A general system error occurred: The source detected that the destination failed to resume".
I've been browsing through the KB and among other I found these who are almost identical with my problem:
- VMware KB: vMotion fails at 82% with the error: General system error occurred: Source detected that destination fail… - but this is not exactly my case since VT is enabled on all hosts and there was no recent firmware upgrade.
- VMware KB: vMotion fails at 82% with the hostd log error: Source detected that destination failed to resume - also does not apply since the VM does not need upgrade nor the UUIDs for the datastores are different on the hosts.
- VMware KB: vMotion fails at 82% with the error: General system error occured - Failed to resume - VMSafe filter are not present
- VMware KB: Performing vMotion fails at 78% with the error: Could not open change tracking file - ctk files are not present.
For reference the logs from the source host:
2015-12-09T19:24:13.151Z [60E03B90 verbose 'Vcsvc.VMotionSrc (1449689051851921)' opID=08AF1D8A-00008FF3-ca-17-d2] Migration changed state from BEGIN to MIGRATING
2015-12-09T19:24:13.151Z [5FC98B90 info 'vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Disconnect check in progress.
2015-12-09T19:24:13.157Z [60E85B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] VMotionCmdCb [1449689051851921] 'start': done
2015-12-09T19:24:14.339Z [5FA81B90 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 28146MHz/253526MB to 27547MHz/253525MB
2015-12-09T19:24:14.797Z [5F640B90 info 'Vimsvc.ha-eventmgr'] Event 175741 : The dvPort 9559 link was down in the vSphere Distributed Switch in ha-datacenter
2015-12-09T19:24:14.817Z [607C3B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Tools are not operations ready
2015-12-09T19:24:14.818Z [607C3B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Tools are not auto-upgrade capable
2015-12-09T19:24:14.819Z [607C3B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Tools guest daemon status changed to: 0
2015-12-09T19:24:14.819Z [607C3B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] guest operations are not ready
2015-12-09T19:24:15.121Z [5FA81B90 info 'Vimsvc.ha-eventmgr'] Event 175742 : The dvPort 9559 was unblocked in the vSphere Distributed Switch in ha-datacenter.
2015-12-09T19:24:15.123Z [5FA81B90 info 'Vimsvc.ha-eventmgr'] Event 175743 : The dvPort 9559 link was up in the vSphere Distributed Switch in ha-datacenter
2015-12-09T19:24:15.127Z [5FA40B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] VMotionLastStatusCb: Failed with error 7: The source detected that the destination failed to resume.
-->
2015-12-09T19:24:15.128Z [5FC98B90 info 'vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Disconnect check in progress.
2015-12-09T19:24:15.140Z [5FA40B90 info 'Hostsvc.HaHost'] vmxSwapEnabled = true vmmOvhd.anonymous: 12096 vmmOvhd.paged: 62659 vmmOvhd.nonpaged: 15543
2015-12-09T19:24:15.140Z [5FA40B90 info 'Hostsvc.HaHost'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2015-12-09T19:24:15.140Z [5FA40B90 info 'Hostsvc.HaHost'] Computed from lib overhead: 113209344 for (memPages: 262144, fbSize: 4096, #vcpus: 1, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 1) vmkOverhead: 5548032
2015-12-09T19:24:15.140Z [5FA40B90 info 'Hostsvc.HaHost'] overhead result (no CBRC): 118755328
2015-12-09T19:24:15.140Z [5FA40B90 info 'Hostsvc.HaHost'] Predicted overhead: 118755328 for (memory: 1073741824, video: 16777216, #vcpus: 1, autodetect: false, enable3D: false, svgaPtFbSize: 16777216, cbrcOverhead: 0
2015-12-09T19:24:15.140Z [5FA40B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Time to gather config: 12 (msecs)
2015-12-09T19:24:15.141Z [60025B90 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 735
2015-12-09T19:24:15.152Z [5FA40B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] VMotionLastStatusCb:: hostlog state changed from emigrating to failure
2015-12-09T19:24:15.152Z [5FA40B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] VMotionResolveCheck: Operation in progress
2015-12-09T19:24:15.152Z [60E03B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Tools guest daemon status changed to: 1
2015-12-09T19:24:15.152Z [60E03B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] guest operations are ready
2015-12-09T19:24:15.153Z [60025B90 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 735 not configured for replication
2015-12-09T19:24:15.158Z [60D40B90 verbose 'Locale'] Default resource used for 'device.VirtualE1000.DistributedVirtualPortBackingInfo.summary' expected in module 'vm'.
2015-12-09T19:24:15.160Z [5FA40B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] VMotionStatusCb: Completed
2015-12-09T19:24:15.160Z [5FA40B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] VMotionResolveCheck: Firing ResolveCb
2015-12-09T19:24:15.161Z [5FA40B90 info 'Vcsvc.VMotionSrc (1449689051851921)'] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
2015-12-09T19:24:15.161Z [608C4B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Tools are operations ready
2015-12-09T19:24:15.161Z [5FDBCB90 verbose 'Locale'] Default resource used for 'device.VirtualE1000.DistributedVirtualPortBackingInfo.summary' expected in module 'vm'.
2015-12-09T19:24:15.161Z [608C4B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Tools are auto-upgrade capable
2015-12-09T19:24:15.162Z [5FA40B90 info 'Vcsvc.VMotionSrc (1449689051851921)'] ResolveCb: Failed with fault: (vmodl.fault.SystemError) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> reason = "The source detected that the destination failed to resume.
--> ",
--> msg = "",
--> }
2015-12-09T19:24:15.162Z [5FA40B90 verbose 'Vcsvc.VMotionSrc (1449689051851921)'] Migration changed state from MIGRATING to DONE
2015-12-09T19:24:15.162Z [5FA40B90 verbose 'Vcsvc.VMotionSrc (1449689051851921)'] Finish called
2015-12-09T19:24:15.164Z [5FA40B90 info 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] State Transition (VM_STATE_EMIGRATING -> VM_STATE_ON)
2015-12-09T19:24:15.165Z [60025B90 verbose 'Hostsvc'] Received state change for VM '735'
2015-12-09T19:24:15.165Z [5FA40B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/VM_FOLDER/c******l3_clone.vmx'] Actual VM overhead: 41926656 bytes
2015-12-09T19:24:15.165Z [60025B90 info 'Guestsvc.GuestFileTransferImpl'] Entered VmPowerStateListener
2015-12-09T19:24:15.165Z [60025B90 info 'Guestsvc.GuestFileTransferImpl'] VmPowerStateListener succeeded
And from the destination host:
2015-12-09T20:19:25.918Z [412FDB90 verbose 'Vcsvc.VMotionDst (1449692364342339)' opID=08AF1D8A-000091F7-55-5c-23] Migration changed state from BEGIN to MIGRATING
2015-12-09T20:19:25.918Z [41157B90 info 'vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Disconnect check in progress.
2015-12-09T20:19:27.906Z [415E7B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] MKS ready for connections: true
2015-12-09T20:19:27.907Z [415E7B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Version status of tools changed to: 2
2015-12-09T20:19:27.907Z [415E7B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Tools version changed to: 9221
2015-12-09T20:19:27.907Z [41157B90 info 'vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Disconnect check in progress.
2015-12-09T20:19:27.907Z [415E7B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Current tools version: 0, tools version updated to 9221
2015-12-09T20:19:27.911Z [415E7B90 warning 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] PopulateCache failed: _diskAccess : false, _storageAccessible : true
2015-12-09T20:19:27.912Z [415E7B90 warning 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] PopulateCache failed: _diskAccess : false, _storageAccessible : true
2015-12-09T20:19:27.920Z [415E7B90 info 'Hostsvc.HaHost'] vmxSwapEnabled = true vmmOvhd.anonymous: 12096 vmmOvhd.paged: 62659 vmmOvhd.nonpaged: 15543
2015-12-09T20:19:27.920Z [415E7B90 info 'Hostsvc.HaHost'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2015-12-09T20:19:27.920Z [415E7B90 info 'Hostsvc.HaHost'] Computed from lib overhead: 113209344 for (memPages: 262144, fbSize: 4096, #vcpus: 1, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 1) vmkOverhead: 5548032
2015-12-09T20:19:27.920Z [415E7B90 info 'Hostsvc.HaHost'] overhead result (no CBRC): 118755328
2015-12-09T20:19:27.920Z [415E7B90 info 'Hostsvc.HaHost'] Predicted overhead: 118755328 for (memory: 1073741824, video: 16777216, #vcpus: 1, autodetect: false, enable3D: false, svgaPtFbSize: 16777216, cbrcOverhead: 0
2015-12-09T20:19:27.920Z [415E7B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Time to gather config: 13 (msecs)
2015-12-09T20:19:27.921Z [415A6B90 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 688
2015-12-09T20:19:27.922Z [415E7B90 warning 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] FetchUpdatedLayout: No cached layout files available. Doing a full fetch
2015-12-09T20:19:27.922Z [415E7B90 warning 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] CannotRetrieveCorefiles: VM disk access is turned off
2015-12-09T20:19:27.932Z [41525B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Retrieved current VM state from foundry 5, 2
2015-12-09T20:19:27.932Z [415A6B90 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 688 not configured for replication
2015-12-09T20:19:27.938Z [42803B90 verbose 'Locale'] Default resource used for 'device.VirtualE1000.DistributedVirtualPortBackingInfo.summary' expected in module 'vm'.
2015-12-09T20:19:27.940Z [414E4B90 verbose 'Locale'] Default resource used for 'device.VirtualE1000.DistributedVirtualPortBackingInfo.summary' expected in module 'vm'.
2015-12-09T20:19:27.940Z [41525B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] VMotionResolveCheck: Operation in progress
2015-12-09T20:19:27.943Z [FFFA6D20 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Handling message _vmx1: The virtual machine cannot be powered on.
-->
2015-12-09T20:19:27.943Z [FFFA6D20 warning 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Failed to find activation record, event user unknown.
2015-12-09T20:19:27.943Z [FFFA6D20 info 'Vimsvc.ha-eventmgr'] Event 84794 : Error message on c******3 on sbcc-hostsrv-2501.root.local in ha-datacenter: The virtual machine cannot be powered on.
2015-12-09T20:19:27.945Z [41157B90 info 'vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Disconnect check in progress.
2015-12-09T20:19:27.945Z [42844B90 info 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Answered question _vmx1
2015-12-09T20:19:27.945Z [42907B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Retrieved current VM state from foundry 5, 2
2015-12-09T20:19:27.945Z [42907B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] VMotionResolveCheck: Operation in progress
2015-12-09T20:19:27.945Z [428C6B90 verbose 'Hostsvc.NetworkProvider'] DVPort 9559 on dvs 2f 36 3b 50 f4 3f 62 6e-fb 7c f7 31 11 61 d7 4d not found
2015-12-09T20:19:27.946Z [428C6B90 verbose 'Hostsvc.NetworkProvider'] DVPort 9559 on dvs 2f 36 3b 50 f4 3f 62 6e-fb 7c f7 31 11 61 d7 4d not found
2015-12-09T20:19:27.946Z [428C6B90 info 'Vimsvc.ha-eventmgr'] Event 84795 : Deleted ports in the vSphere Distributed Switch in ha-datacenter.
2015-12-09T20:19:27.946Z [424C1B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Handling message _vmx2: The VM failed to resume on the destination during early power on.
-->
2015-12-09T20:19:27.947Z [424C1B90 warning 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Failed to find activation record, event user unknown.
2015-12-09T20:19:27.947Z [41157B90 info 'vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Disconnect check in progress.
2015-12-09T20:19:27.947Z [424C1B90 info 'Vimsvc.ha-eventmgr'] Event 84796 : Error message on c******3 on sbcc-hostsrv-2501.root.local in ha-datacenter: The VM failed to resume on the destination during early power on.
-->
2015-12-09T20:19:27.948Z [415E7B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Retrieved current VM state from foundry 5, 2
2015-12-09T20:19:27.948Z [415E7B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] VMotionResolveCheck: Operation in progress
2015-12-09T20:19:27.948Z [427C2B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] VMotionLastStatusCb: Failed with error 7: The VM failed to resume on the destination during early power on.
-->
2015-12-09T20:19:27.948Z [41157B90 info 'vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Disconnect check in progress.
2015-12-09T20:19:27.948Z [427C2B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] VMotionResolveCheck: Operation in progress
2015-12-09T20:19:27.949Z [427C2B90 info 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Answered question _vmx2
2015-12-09T20:19:27.949Z [427C2B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] Retrieved current VM state from foundry 5, 2
2015-12-09T20:19:27.949Z [427C2B90 verbose 'Vmsvc.vm:/vmfs/volumes/52396d12-a22179b6-ad8a-0025b520a48f/c******3/c******3_clone.vmx'] VMotionResolveCheck: Operation in progress
2015-12-09T20:19:27.950Z [412BCB90 info 'Hostsvc.HaHost'] vmxSwapEnabled = true vmmOvhd.anonymous: 12096 vmmOvhd.paged: 62659 vmmOvhd.nonpaged: 15543
I will appreciate any ideas,
Rares