failed live migration to a different host on cloudstack 4.19.1

[Chih80]

problem

Live migration keeps on failing when trying to move VM to another host. Same result when tried on multiple host.

versions

cloudstack version 4.19.1 on ubuntu 22.04
kvm host on ubuntu 20,04
NFS on secondary storage

The steps to reproduce the bug

1. Selected instance
2. click migration to another host
3. selected host and migrate with storage on same host
4. After a few seconds, error comes back with:
5. Migration failed.
   Failed to migrate VM [VM instance {"id":103,"instanceName":"i-5-103-VM","type":"User","uuid":"5edeefc5-462d-4dda-b082-4358ec49486d"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "102", from: "1", to:"10"}] and VM to host [{vm: "103", from: "1", to:"27"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to stat /var/lib/libvirt/images/870ee0f4-212c-4116-8bf1-99b98985b9a7: No such file or directory].].
   ...

What to do about it?

No response

[DaanHoogland]

@Chih80 , can you investigate the MS - and agent logs for around the time of the error?

Are there any ERROR logs or stack traces preceding the error above?

edit: also please check the versions of libvirt qemu and the cloudstack agents on the hosts, to see if these match.