Failed to add additional management server: No bean named 'DefaultConfigResources' available

[MejdiB]

problem

When trying to add an additional management server without DB replication, the UI fails with HTTP ERROR 503 Service Unavailable due to the following Stack Trace:

module start failure of module [nested-cloudstack] was due to:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.19.2.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]: Cannot resolve reference to bean 'DefaultConfigResources' while setting bean property 'locations'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'DefaultConfigResources' available
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:342)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:113)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1707)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1452)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:171)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:748)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:173)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:142)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:273)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:278)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:278)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:278)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:278)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:261)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:130)
        at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:78)
        at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
        at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
        at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
        at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
        at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
        at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:423)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
        at org.eclipse.jetty.server.Server.doStart(Server.java:387)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:203)
        at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:114)
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'DefaultConfigResources' available
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:874)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1358)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:309)
        ... 57 more

versions

• ACS 4.19.2
• AlmaLinux 8.10

Please note that this behaviour does not occur on ACS 4.17 (with exact same installation steps)

The steps to reproduce the bug

1. Install cloudstack-management on AlmaLinux 8.10 fresh installation with v4.19.2
2. Configure IPv4 properly via nmtui and get the organization IP address on the machine
3. Check rpc bind "systemctl status rpcbind"
4. Perform "cloudstack-setup-databases cloud:@"
5. Configure firewalld and selinux (permissive)
6. cloudstack-setup-management

The db.properties file is setup up correctyl, also the correct organization IPv4 for the management server is derived with cloudstack-setup-databases

What to do about it?

Please provide a workaround and fix this issue

[weizhouapache]

please check the logs if there are some SQL exceptions or other exceptions
you can search for "Server configuration file found" and check logs from there

[apraba05]

I am experiencing the same issue. I changed the Cluster IP on the db.properties of the files according to one of the previous tickets. This is from a fresh installation and I am getting a 503.

[Indrasura20]

I had this same issue and was able to figure this out:
Saw this initial error:
Caused by: com.cloud.utils.exception.CloudRuntimeException: The script injectkeys.sh failed to run successfully : Files /var/cloudstack/management/.ssh/id_rsa and /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud differ

Ran this command, restarted cloudstack-management and it works now:
cp /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud /var/cloudstack/management/.ssh/id_rsa

[MejdiB]

@weizhouapache There is no "SQL"* error in the logs. However, I used this workaroudn:

1. Remove cloudstack-management version 4.19
2. Edit cloudstack.repo to retrieve 4.17
3. Install ACS 4.17.
4. Build database etc. and setup cloudstack-management (single command line)
5. After that the ACS server would not come up since version discrepancy from DB version 4.19. and 4.17 ACS
6. Upgrade cloudstack-management to version 4.19
7. Restart cloudstack-management and now it works

[weizhouapache]

@weizhouapache There is no "SQL"* error in the logs. However, I used this workaroudn:

1. Remove cloudstack-management version 4.19
2. Edit cloudstack.repo to retrieve 4.17
3. Install ACS 4.17.
4. Build database etc. and setup cloudstack-management (single command line)
5. After that the ACS server would not come up since version discrepancy from DB version 4.19. and 4.17 ACS
6. Upgrade cloudstack-management to version 4.19
7. Restart cloudstack-management and now it works

@MejdiB
thanks for the sharing

IMHO, it is a strange issue

would it be possible to upload the log ?

[MejdiB]

@weizhouapache I try to shrink this, because this above error snippets is generated in a loop-like behaviour.

[weizhouapache]

@weizhouapache I try to shrink this, because this above error snippets is generated in a loop-like behaviour.

maybe just the lines between Server configuration file found and first org.springframework.beans.factory.BeanCreationException

[vishesh92]

@MejdiB Can you check if you are able to connect to the database from the new management server? It's hard to pin point the exact error without the complete logs.