Secondry Storage NFS v3

[UAnton]

problem

KVM nodes can't deploy Service VMs.

2025-03-11 21:32:30,235 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-28:[ctx-5b29434d, job-113/job-141, ctx-9e239170]) (logid:cbf99996) Unable to create volume [{"name":"ROOT-38","uuid":"32eba291-1115-4dfe-b90c-a64de46a2de4"}] due to [com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 10.100.7.4:/secondary/template/tmpl/1/3 /mnt/3571e6e5-e631-3d46-8a7e-b2b66f8f4bc4) unexpected exit status 32: mount.nfs: Protocol not supported
].
2025-03-11 21:32:30,235 WARN  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-28:[ctx-5b29434d, job-113/job-141, ctx-9e239170]) (logid:cbf99996) Unable to contact resource. com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create volume [{"name":"ROOT-38","uuid":"32eba291-1115-4dfe-b90c-a64de46a2de4"}] due to [com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec 10.100.7.4:/secondary/template/tmpl/1/3 /mnt/3571e6e5-e631-3d46-8a7e-b2b66f8f4bc4) unexpected exit status 32: mount.nfs: Protocol not supported
].
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1797)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1944)
        at jdk.internal.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy265.prepare(Unknown Source)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1265)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5467)
        at jdk.internal.reflect.GeneratedMethodAccessor207.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5591)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

versions

ACS 4.20
Alma Linux 9
VNXe3200

The steps to reproduce the bug

...

What to do about it?

No response

[nischalnischal2020]

HI @UAnton ,

This error occurs when there is a mismatch in the NFS versions that cloudstack is trying to mount and the version that you are using on the nfs server.

Please check the nfs version mentioned in your Configuration "secstorage.nfs.version"

Also check if the nfs server is running on a higher version only

you can check the version of nfs on the server using

rpcinfo -p

Please make sure that version matches in cloudstack and nfs server

[UAnton]

@nischalnischal2020 Hi,
nfs-server version is 3.

secstorage.nfs.version

[nischalnischal2020]

Hi @UAnton

Can you please manually mount the nfs share on the KVM hosts and check if it works?