Slow container startup (1m 32s) when using custom PUID/PGID

[tomaskir]

Support guidelines

• I've read the support guidelines

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Greetings,

When using my own PUID/PGID, the Libre containers (server, dispatcher, syslog, snmptrap) take 1 minute 30 seconds to start each.

services:
  nms.x.internal:
    image: librenms/librenms:26.2.0
...
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
...

The root cause of the issue is here:

docker/rootfs/etc/cont-init.d/03-config.sh

Line 222 in df201a3

find /data/logs ${LIBRENMS_PATH}/composer* ${LIBRENMS_PATH}/config.d ${LIBRENMS_PATH}/bootstrap ${LIBRENMS_PATH}/logs ${LIBRENMS_PATH}/storage ${LIBRENMS_PATH}/vendor \( ! -user librenms -o ! -group librenms \) -exec chown librenms:librenms {} +

The issue is in the /opt/librenms/vendor, which has many files:

3ae6986beb62:/etc/cont-init.d# find /opt/librenms/vendor | wc -l
11683
3ae6986beb62:/etc/cont-init.d# 

This command takes excessively long to run:

docker compose run -it --rm --entrypoint bash nms.x.internal-syslog
...
7f53b6cc3442:/opt/librenms# 
7f53b6cc3442:/opt/librenms# cd /etc/cont-init.d/
7f53b6cc3442:/etc/cont-init.d# bash 00-fix-logs.sh 
7f53b6cc3442:/etc/cont-init.d# bash 01-fix-uidgid.sh 
Switching to PGID 805...
Switching to PUID 805...
7f53b6cc3442:/etc/cont-init.d# bash 02-fix-perms.sh 
Fixing perms...
7f53b6cc3442:/etc/cont-init.d# time find /opt/librenms/vendor \( ! -user librenms -o ! -group librenms \) -exec chown librenms:librenms {} +

real    1m32.008s
user    0m0.069s
sys     0m3.614s
7f53b6cc3442:/etc/cont-init.d# 

This could be replaced with a simple:

chown -R --from=root:root librenms:librenms ${LIBRENMS_PATH}/vendor

And removing the ${LIBRENMS_PATH}/vendor from the existing find ... -exec call.

This would have the same effect, but take only seconds. Here is a fresh container run with the updated command:

docker compose run -it --rm --entrypoint bash nms.x.internal-syslog
...
3ae6986beb62:/opt/librenms# cd /etc/cont-init.d/
3ae6986beb62:/etc/cont-init.d# bash 00-fix-logs.sh 
3ae6986beb62:/etc/cont-init.d# bash 01-fix-uidgid.sh 
Switching to PGID 805...
Switching to PUID 805...
3ae6986beb62:/etc/cont-init.d# bash 02-fix-perms.sh 
Fixing perms...
3ae6986beb62:/etc/cont-init.d# time chown -R --from=root:root librenms:librenms /opt/librenms/vendor

real    0m0.282s
user    0m0.024s
sys     0m0.257s
3ae6986beb62:/etc/cont-init.d# 

This would take down container startup time from 1 minute 30+ seconds to almost instant.

Expected behaviour

Container will start fast.

Actual behaviour

Container takes 1 minute 30+ seconds to start.

Steps to reproduce

Run container with custom PUID/PGID:

services:
  nms.x.internal:
    image: librenms/librenms:26.2.0
...
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
...

Docker info

Client: Docker Engine - Community
 Version:    29.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Docker Compose config

Logs

not applicable

Additional info

No response

[murrant]

Cool, that's what it was before someone said it was slow and changed it to what it is now...

[tomaskir]

@murrant I do see this was discussed in #524, and then changed in #540.

But you can see from my time runs above that (at least for me) chown -R --from=root:root librenms:librenms ... is MUCH faster.

Before the change the stack took 4+ minutes to start (main server container start 1 min 40 sec, then another 2 minutes for dispatcher, syslog and snmptrap start due to dependencies in depends_on).

After I patched and mounted a modified /etc/cont-init.d/03-config.sh my entire Libre stack starts in literally 10 seconds.

[murrant]

@tomaskir why is the current script running anything? It should only change the owner if the owner is incorrect. (which should hopefully be one time)

[tomaskir]

@murrant only "./librenms:/data" is mounted. docker compose down && docker compose up will cause the script to need to fix permission again in ${LIBRENMS_PATH} / /opt/librenms.

Or a version update, basically anything that will recreate the container.

[murrant]

Oh because the specified user doesn't match the file owner in the container image.

There has to be a better solution.

[tomaskir]

@murrant Yeah, in general having to fix permission on 12.000 files on every container recreate is not great. BUT the problem before was that the chown that was used tried to change permissions for all those files on every start. In the mentioned discussion and the following PR, it was changed to the find that only changes permissions on files which were NOT already fixed.

That's good, but the current code (the find) in effect generates a chown command with literally 12.000 arguments on container recrete. This is why it takes 1 min 30+ seconds.

My suggestion is to change to chown -R --from=root:root librenms:librenms ... which BOTH only changes the files if the owner is not already librenms AND is much faster if it needs to change all the files (since it handles iterating over the 12.000 files in chown directly rather than crating a chown with 12.00 arguments from the find).

Hope that makes sense.

[murrant]

The goal of the find was to improve the speed of startups after the first.

What I was wondering was if there is a solution where we can remove the chown entirely.

[tomaskir]

The goal of the find was to improve the speed of startups after the first.

I get that, but the same can be done with chown -R --from=root:root librenms:librenms .... It has the same benefits, but no penalty on the first startup. As I mentioned above, currently the first startup generates a chown with 12.000 arguments as a result of the find.

You can see (and test yourself) the difference with the test cases provided in my first post.

What I was wondering was if there is a solution where we can remove the chown entirely.

This would indeed be ideal.