Replies: 4 comments 15 replies
-
|
@zidz Can you share the managment-server.log from all the management servers covering the time of your tests. Also, share the name of the VM ( which was manually stopped) |
Beta Was this translation helpful? Give feedback.
-
|
I've added links to the logs, and they are somewhat redacted, the instance name is "test", the internal name for it is "i-6-215-VM". cloudstack1.txt |
Beta Was this translation helpful? Give feedback.
-
|
@zidz, can you try to hard power down your KVM host and see if those VMs running on this host will failover? Also ensure that your VM is created with "HA enabled=true". In my experience, if you shutdown gracefully, these VMs will not failover. |
Beta Was this translation helpful? Give feedback.
-
|
@zidz I agree with what @chunkyen is suggesting here. Based on what we see in the logs ( snippet below). Cloudstack has detected the VM is not running and has initiated HA worker threads as well. However the VM is not started else where as the host is still running and reachable 2026-01-27 12:01:05,712 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-837f2660, work-554]) (logid:8894ff2b) VM VM instance {"id":215,"instanceName":"i-6-215-VM","state":"Running","type":"User","uuid":"16f60c4c-4cb5-46be-8ddc-62c774f3e502"} is found to be alive by KVMInvestigator 2026-01-27 23:02:53,845 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-3:[ctx-85d9be02, work-602]) (logid:f52990b8) VM VM instance {"id":215,"instanceName":"i-6-215-VM","state":"Running","type":"User","uuid":"16f60c4c-4cb5-46be-8ddc-62c774f3e502"} is found to be alive by KVMInvestigator From what I see in the code ( with my limited ability to read code and using gemini), the above lines indicate cloudstack assumes the VM to be up as the host is up and is writing its heartbeat to storage. |
Beta Was this translation helpful? Give feedback.
-
|
There is this setting and this should do what @zidz is asking for. |
Beta Was this translation helpful? Give feedback.
-
|
@chunkyen yes came across this. I tested on 4.22 with the same setting enabled , but the VM doesn't start on out-band-of-stop >>I do remember this use to work before though. |
Beta Was this translation helpful? Give feedback.
-
|
Issues reported link |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
A note, the Event log for the VM in question looks like this after it's killed (-9), and it's keep doing that in the same frequence until it's stopped in the UI: |
Beta Was this translation helpful? Give feedback.
-
|
Currently there is a issue with HOST HA and VM HA sync, investigation is going to find the root cause I would recommend to use only VM HA feature for now |
Beta Was this translation helpful? Give feedback.
-
|
I think the OP is only talking about VM HA. |
Beta Was this translation helpful? Give feedback.
-
|
KVM HA is a bit confuring but you might be right. That said, one cannot expect ACS to be sure if a VM is down if the host is not fenced and a power report from the host shows the VM as down. There is an expectation that a host being down is taken for a fact by ACS, but that can lead to split brain scenarios and consequently to data loss. @zidz , KVM HA is not a term that is used in cloudstack, can you clarify your expectations? (so we can assess better if these are realistic ;) |
Beta Was this translation helpful? Give feedback.
-
|
I'm very sorry, I mean VM HA as @chunkyen suggest :-D @DaanHoogland , Think of a scenario where the host is UP, and the cloudstack agent is running. If I forcefully kill a VM (eg. kill -9 $VM_PID) there should be a possibility for the agent to realize that this VM is gone and should tell the scheduler to start it again, preferably on the least loaded host. |
Beta Was this translation helpful? Give feedback.
-
|
@kiranchavala , As my current hardware lack IPMI I only use VM HA. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I do not know if I have unrealistic expectations of Cloudstack 4.22 in regards to VM HA.
I'm running Cloudstack on hardware with no IPMI and expect some failover when a KVM instance is manually stopped on the host, is unexpectedly killed or dies. Cloudstack lists the VM as running and nothing happens, the VM will not be restarted to any extent. What I expect is that the VM should be detected as not running when it's expected to and should be restarted, preferably on another host. Note that I've waited over an hour to give all checks time to trigger a restart of a killed VM.
The current setup is running on NFS for primary and secondary right now. I expect the next cluster to have its primary on Ceph RBD and secondary on NFS hosted by Ceph.
I've read the HA documentation and I've deactivated all Host HA, as this else will be taken into account before handling the VM HA (As far as I understand). Checked that the files in the KVMHA are updated with timestamps correctly. NTP is running correctly. All VM's do have HA activated, all compute offerings I use have HA enabled, I even forced HA for VM's in the Global configuration.
Please tell me how VM HA is intended to work, if my expectations are off and I would be super grateful for VM HA test examples I can do to trigger the VM HA functionality to get the correct expectations of the VM HA functionality.
I've read earlier discussions here in the Q&A section and other sources to try to solve this HA puzzle, but now I'm here asking for assistance..
Beta Was this translation helpful? Give feedback.
All reactions