[Bug]: running with bind mounted ca dir breaks the container

[rwaffen]

Is this a critical security issue?

• This is not a security issue.

Describe the Bug

cd /home/rwaffen/Development/Voxpupuli/Containers/Server
❯ tree
.
└── ca

❯  podman run --rm -it -v $PWD/ca:/etc/puppetlabs/puppetserver/ca ghcr.io/openvoxproject/openvoxserver:8.12.1-main

Trying to pull ghcr.io/openvoxproject/openvoxserver:8.12.1-main...
Getting image source signatures
Copying blob 29dd6a22d697 done   |
Copying blob 1bf09a6aae3d done   |
Copying blob eb94a96e43c4 done   |
Copying blob 1fe57b047861 done   |
Copying blob 817807f3c64e done   |
Copying blob 8cf4b57493f6 done   |
Copying blob 1ffc8b025703 done   |
Copying blob 59706481acc6 done   |
Copying blob b0b597d23b35 done   |
Copying blob 492c391f41ff done   |
Copying blob 0c91d6dd3578 done   |
Copying blob 953ee3748d0e done   |
Copying config 4e5e58f124 done   |
Writing manifest to image destination
Entrypoint PID 2
Running /container-entrypoint.d/20-use-templates-initially.sh
Copying template puppetdb.conf from /var/tmp/puppet
Upgrading /opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems
Running /container-entrypoint.d/30-ensure-config.sh
Running /container-entrypoint.d/40-update-puppetdb-conf.sh
Running /container-entrypoint.d/50-set-certname.sh
Running /container-entrypoint.d/55-set-masterport.sh
Running /container-entrypoint.d/56-set-environmentpath.sh
Running /container-entrypoint.d/57-set-hiera_config.sh
Running /container-entrypoint.d/60-setup-autosign.sh
Running /container-entrypoint.d/70-set-dns-alt-names.sh
Running /container-entrypoint.d/83-environment-cache.sh
Settings environment_timeout to unlimited
Running /container-entrypoint.d/84-enable_graphite.sh
Running /container-entrypoint.d/85-setup-storeconfigs.sh
Running /container-entrypoint.d/88-enable-cache-delete-api.sh
Running /container-entrypoint.d/89-csr_attributes.sh
CSR Attributes: {}
Running /container-entrypoint.d/90-ca.sh
Generation succeeded. Find your files in /etc/puppetlabs/puppetserver/ca
Running /container-entrypoint.d/99-log-config.sh
System configuration values:
* HOSTNAME: '183c47e807d4'
* hostname -f: '183c47e807d4'
* CERTNAME: unset, try to use the oldest certificate in the certs directory, because this might be the one that was used initially.
* OPENVOXSERVER_PORT: '8140'
* Certname: '183c47e807d4.fritz.box.pem'
* DNS_ALT_NAMES: ''
* SSLDIR: '/etc/puppetlabs/puppet/ssl'
CA Certificate:
subject=CN = "Puppet CA generated on 183c47e807d4 at 2026-04-02 17:09:51 +0000"
issuer=CN = Puppet Root CA: 2e35124fbe9f80
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier:
                64:29:68:02:49:8E:91:E2:D2:B2:61:3E:9A:2D:11:AC:6B:BB:D8:F6
            Netscape Comment:
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier:
                9B:B3:2F:D0:E9:65:6C:5E:A7:DD:DA:C8:9F:CB:3D:13:F1:2E:85:7C
Certificate 183c47e807d4.fritz.box.pem:
subject=CN = 183c47e807d4.fritz.box
issuer=CN = "Puppet CA generated on 183c47e807d4 at 2026-04-02 17:09:51 +0000"
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            Netscape Comment:
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier:
                64:29:68:02:49:8E:91:E2:D2:B2:61:3E:9A:2D:11:AC:6B:BB:D8:F6
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Subject Key Identifier:
                EB:6F:1A:85:8C:23:01:45:7A:A9:40:65:E2:45:5C:B0:23:2B:C2:09
            1.3.6.1.4.1.34380.1.3.39:
                ..true
            X509v3 Subject Alternative Name:
                DNS:puppet, DNS:183c47e807d4.fritz.box
Starting Puppetserver
2026-04-02 17:09:56,584 INFO  [p.t.s.w.jetty10-core] Removing buggy security provider SunPKCS11 version 21
2026-04-02 17:09:56,770 INFO  [p.t.s.s.scheduler-service] Initializing Scheduler Service
2026-04-02 17:09:56,783 INFO  [o.q.i.StdSchedulerFactory] Using default implementation for ThreadExecutor
2026-04-02 17:09:56,788 INFO  [o.q.c.SchedulerSignalerImpl] Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2026-04-02 17:09:56,790 INFO  [o.q.c.QuartzScheduler] Quartz Scheduler v2.5.2 created.
2026-04-02 17:09:56,790 INFO  [o.q.s.RAMJobStore] RAMJobStore initialized.
2026-04-02 17:09:56,790 INFO  [o.q.c.QuartzScheduler] Scheduler meta-data: Quartz Scheduler (v2.5.2) '7e3e79b5-c0f9-4ce5-871b-88cdaa9e40d9' with instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2026-04-02 17:09:56,790 INFO  [o.q.i.StdSchedulerFactory] Quartz scheduler '7e3e79b5-c0f9-4ce5-871b-88cdaa9e40d9' initialized from an externally provided properties instance.
2026-04-02 17:09:56,790 INFO  [o.q.i.StdSchedulerFactory] Quartz scheduler version: 2.5.2
2026-04-02 17:09:56,790 INFO  [o.q.c.QuartzScheduler] Scheduler 7e3e79b5-c0f9-4ce5-871b-88cdaa9e40d9_$_NON_CLUSTERED started.
2026-04-02 17:09:56,791 INFO  [p.t.s.w.jetty10-service] Initializing web server(s).
2026-04-02 17:09:56,799 INFO  [p.t.s.s.status-service] Registering status callback function for service 'puppet-profiler', version 8.12.1
2026-04-02 17:09:56,800 INFO  [p.s.j.jruby-puppet-service] Initializing the JRuby service
2026-04-02 17:09:56,803 INFO  [p.s.j.jruby-pool-manager-service] Initializing the JRuby service
2026-04-02 17:09:56,805 INFO  [p.s.j.jruby-puppet-service] JRuby version info: jruby 9.4.12.1 (3.1.4) 2025-05-07 64b3479ca4 OpenJDK 64-Bit Server VM 21.0.10+7-Ubuntu-124.04 on 21.0.10+7-Ubuntu-124.04 +jit [x86_64-linux]
2026-04-02 17:09:56,808 INFO  [p.s.j.i.jruby-internal] Creating JRubyInstance with id 1.
2026-04-02 17:09:59,487 INFO  [puppetserver] Puppet Puppet settings initialized; run mode: server
2026-04-02 17:10:00,393 INFO  [p.s.j.i.jruby-agents] Finished creating JRubyInstance 1 of clojure.core$count@4625a3e2
2026-04-02 17:10:00,396 INFO  [p.s.c.puppet-server-config-core] Initializing webserver settings from core Puppet
2026-04-02 17:10:00,403 ERROR [p.t.internal] Error during service init!!!
clojure.lang.ExceptionInfo: throw+: {:kind :puppetlabs.kitchensink.core/io-error, :msg "Parent directory '/etc/puppetlabs/puppetserver/ca' is not writable"}
	at slingshot.support$stack_trace.invoke(support.clj:201)
	at puppetlabs.kitchensink.core$mkdirs_BANG_$fn__1728.invoke(core.clj:195)
	at puppetlabs.kitchensink.core$mkdirs_BANG_.invokeStatic(core.clj:186)
	at puppetlabs.kitchensink.core$mkdirs_BANG_.invoke(core.clj:165)
	at puppetlabs.puppetserver.certificate_authority$fn__40608$ensure_directories_exist_BANG___40613$fn__40614.invoke(certificate_authority.clj:1954)
	at puppetlabs.puppetserver.certificate_authority$fn__40608$ensure_directories_exist_BANG___40613.invoke(certificate_authority.clj:1948)
	at puppetlabs.puppetserver.certificate_authority$fn__40663$initialize_BANG___40668$fn__40669.invoke(certificate_authority.clj:1977)
	at puppetlabs.puppetserver.certificate_authority$fn__40663$initialize_BANG___40668.invoke(certificate_authority.clj:1970)
	at puppetlabs.services.ca.certificate_authority_service$reify__46440$service_fnk__5265__auto___positional$reify__46458.init(certificate_authority_service.clj:64)
	at puppetlabs.trapperkeeper.services$fn__5089$G__5081__5092.invoke(services.clj:7)
	at puppetlabs.trapperkeeper.services$fn__5089$G__5080__5096.invoke(services.clj:7)
	at puppetlabs.trapperkeeper.internal$fn__14305$run_lifecycle_fn_BANG___14312$fn__14313.invoke(internal.clj:242)
	at puppetlabs.trapperkeeper.internal$fn__14305$run_lifecycle_fn_BANG___14312.invoke(internal.clj:225)
	at puppetlabs.trapperkeeper.internal$fn__14335$run_lifecycle_fns__14340$fn__14341.invoke(internal.clj:275)
	at puppetlabs.trapperkeeper.internal$fn__14335$run_lifecycle_fns__14340.invoke(internal.clj:252)
	at puppetlabs.trapperkeeper.internal$fn__14970$build_app_STAR___14979$fn$reify__14991.init(internal.clj:660)
	at puppetlabs.trapperkeeper.internal$fn__15021$boot_services_for_app_STAR__STAR___15028$fn__15029$fn__15031.invoke(internal.clj:698)
	at puppetlabs.trapperkeeper.internal$fn__15021$boot_services_for_app_STAR__STAR___15028$fn__15029.invoke(internal.clj:697)
	at puppetlabs.trapperkeeper.internal$fn__15021$boot_services_for_app_STAR__STAR___15028.invoke(internal.clj:691)
	at clojure.core$partial$fn__5931.invoke(core.clj:2647)
	at puppetlabs.trapperkeeper.internal$fn__14381$initialize_lifecycle_worker__14392$fn__14393$fn__14556$state_machine__11619__auto____14581$fn__14584.invoke(internal.clj:295)
	at puppetlabs.trapperkeeper.internal$fn__14381$initialize_lifecycle_worker__14392$fn__14393$fn__14556$state_machine__11619__auto____14581.invoke(internal.clj:295)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:58)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:57)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:62)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:60)
	at clojure.core.async$ioc_alts_BANG_$fn__11851.invoke(async.clj:453)
	at clojure.core.async$do_alts$fn__11786$fn__11789.invoke(async.clj:320)
	at clojure.core.async.impl.channels$appm$fn__6412.invoke(channels.clj:36)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2026-04-02 17:10:00,406 INFO  [p.t.internal] Beginning shutdown sequence
2026-04-02 17:10:00,407 INFO  [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopping metrics sampler job
2026-04-02 17:10:00,407 INFO  [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopped metrics sampler job
2026-04-02 17:10:00,408 INFO  [p.s.c.certificate-authority-service] Stopping CA service
2026-04-02 17:10:00,409 INFO  [p.s.j.i.jruby-agents] Draining JRuby pool.
2026-04-02 17:10:00,410 INFO  [p.s.j.i.jruby-agents] Borrowed all JRuby instances, proceeding with cleanup.
2026-04-02 17:10:00,418 INFO  [p.s.j.i.jruby-internal] Cleaned up old JRubyInstance with id 1.
2026-04-02 17:10:00,419 INFO  [p.s.j.i.jruby-agents] Finished draining pool.
2026-04-02 17:10:00,420 INFO  [p.t.s.w.jetty10-service] Shutting down web server(s).
2026-04-02 17:10:00,421 INFO  [p.t.s.w.filesystem-watch-service] Shutting down watcher service
2026-04-02 17:10:00,422 INFO  [p.t.s.w.filesystem-watch-core] Closing watcher puppetlabs.trapperkeeper.services.watcher.filesystem_watch_core.WatcherImpl@4e2ff515
2026-04-02 17:10:00,422 INFO  [p.t.s.w.filesystem-watch-service] Done shutting down watcher service
2026-04-02 17:10:00,423 INFO  [p.t.s.s.scheduler-service] Shutting down Scheduler Service
2026-04-02 17:10:00,423 INFO  [o.q.c.QuartzScheduler] Scheduler 7e3e79b5-c0f9-4ce5-871b-88cdaa9e40d9_$_NON_CLUSTERED shutting down.
2026-04-02 17:10:00,423 INFO  [o.q.c.QuartzScheduler] Scheduler 7e3e79b5-c0f9-4ce5-871b-88cdaa9e40d9_$_NON_CLUSTERED paused.
2026-04-02 17:10:00,790 INFO  [o.q.c.QuartzScheduler] Scheduler 7e3e79b5-c0f9-4ce5-871b-88cdaa9e40d9_$_NON_CLUSTERED shutdown complete.
2026-04-02 17:10:00,791 INFO  [p.t.s.s.scheduler-service] Scheduler Service shutdown complete.
2026-04-02 17:10:00,794 INFO  [p.t.internal] Finished shutdown sequence
Execution error (ExceptionInfo) at slingshot.support/stack-trace (support.clj:201).
throw+: {:kind :puppetlabs.kitchensink.core/io-error, :msg "Parent directory '/etc/puppetlabs/puppetserver/ca' is not writable"}

Full report at:
/tmp/clojure-2975824286235042511.edn

Expected Behavior

container starts and runs without errors

Steps to Reproduce

• cd /some/dir
• mkdir ca
• podman run --rm -it -v $PWD/ca:/etc/puppetlabs/puppetserver/ca ghcr.io/openvoxproject/openvoxserver:8.12.1-main

Environment

podman version 5.8.1
container tag 8.12.1-main
non selinux system

[rwaffen]

after running into this, i have those file in the ca dir

❯ tree -hug
[rwaffen   rwaffen      4]  .
└── [rwaffen   rwaffen    244]  ca
    ├── [100998   100998   1.9K]  ca_crl.pem
    ├── [100998   100998   3.8K]  ca_crt.pem
    ├── [100998   100998   3.2K]  ca_key.pem
    ├── [100998   100998    800]  ca_pub.pem
    ├── [100998   100998   1.9K]  infra_crl.pem
    ├── [100998   100998      1]  infra_inventory.txt
    ├── [100998   100998      1]  infra_serials
    ├── [100998   100998     80]  inventory.txt
    ├── [100998   100998   3.2K]  root_key.pem
    ├── [100998   100998      4]  serial
    └── [100998   100998     52]  signed  [error opening dir]

[rwaffen]

ca dir inside the container

podman run --rm -it -v $PWD/ca:/etc/puppetlabs/puppetserver/ca --entrypoint bash ghcr.io/openvoxproject/openvoxserver:8.12.1-main

root@4f7b4f20ae5b:/etc/puppetlabs/puppetserver# tree -hug
[puppet   puppet      4]  .
|-- [root     root      244]  ca
|   |-- [puppet   puppet   1.9K]  ca_crl.pem
|   |-- [puppet   puppet   3.8K]  ca_crt.pem
|   |-- [puppet   puppet   3.2K]  ca_key.pem
|   |-- [puppet   puppet    800]  ca_pub.pem
|   |-- [puppet   puppet   1.9K]  infra_crl.pem
|   |-- [puppet   puppet      1]  infra_inventory.txt
|   |-- [puppet   puppet      1]  infra_serials
|   |-- [puppet   puppet     80]  inventory.txt
|   |-- [puppet   puppet   3.2K]  root_key.pem
|   |-- [puppet   puppet      4]  serial
|   `-- [puppet   puppet     52]  signed
|       `-- [puppet   puppet   2.0K]  183c47e807d4.fritz.box.pem
|-- [root     root       58]  conf.d
|   |-- [root     root     9.0K]  auth.conf
|   |-- [root     root      694]  ca.conf
|   |-- [root     root      196]  global.conf
|   |-- [root     root     1.7K]  metrics.conf
|   |-- [root     root       42]  product.conf
|   |-- [root     root     3.9K]  puppetserver.conf
|   |-- [root     root      875]  web-routes.conf
|   `-- [root     root      150]  webserver.conf
|-- [root     root      434]  logback.xml
|-- [root     root      302]  request-logging.xml
`-- [root     root       12]  services.d
    `-- [root     root      428]  ca.cfg

5 directories, 22 files

[rwaffen]

on a normal startup the steps after the error should be those. so this is breaking somehow

2026-04-02 17:34:22,359 INFO  [p.p.certificate-authority] CA already initialized for SSL
2026-04-02 17:34:22,359 INFO  [p.s.c.certificate-authority-service] CA Service adding a ring handler
2026-04-02 17:34:22,375 INFO  [p.t.s.s.status-service] Registering status callback function for service 'ca', version 8.12.1
...

[sebastianrakel]

this was the bug i was talking about with you yesterday