Reduce memory footprint

[rgaudin]

Based on #45, we target the following devices:

• Pi 3A+ (512MB)
• Pi 3B (1GB)
• Pi 3B+ (1GB)
• Pi 4B (1GB, 2GB, 4GB, 8GB)
• Pi 400 (4GB)
• Pi Zero 2W (512MB)

Of this list, 2 devices have 512MB RAM: Pi 3A+ (arguably old) and Zero 2W which is a recent device that respond to specific use cases.

An off the shelf base-image currently leaves only 128MB free RAM for all applications. That's not enough to support our baseline of tools (captive-portal, home, metrics) and leave enough for content apps. Reminder: we're using Docker so a lot of stuff are duplicated.

A Quick look at htop shows a great deal of RAM is used by thttpd so maybe that's the main culprit here; although that's strange since its job is so limited. Given it's only used in the base-image to serve as a demo, we can't really count it as base-image usage.

More investigation is needed with actual data.

[kelson42]

@rgaudin I'm worried to read that metrics would be a mandatory package. I think it should not.

[rgaudin]

@rgaudin I'm worried to read that metrics would be a mandatory package. I think it should not.

You misread. I mentioned “baseline of tools” listing it in opposition to content. Neither those kiwix tools nor content are of concerned in this repo. I just want to emphasis that the bare system is already using a lot of RAM

[rgaudin]

Now that we're based off bookworm, podman is available in a recent-enough (4.3.1) package. We may look into #7 again if docker turns out to be an issue on Pi Zero2W

[kelson42]

@rgaudin Being impatient. Anything stopping us to make right now a wikimed image and test on this HW?

[rgaudin]

@rgaudin Being impatient. Anything stopping us to make right now a wikimed image and test on this HW?

I don't have time to investigate but every time I booted on Pi2ZW with a working docker-compose (and the rest), it froze. When the compose failed to start (missing binding folder for instance) the system worked. I have bigger fishes to fry first but I'll come to this and post updates.

[rgaudin]

This is not looking very good.

I've looked into switching our docker stack to podman but ran into issues around networking.
Podman's typical use case is very different from ours and nobody appears to be using it on arm64.
Still a good option to consider in the future but given how fast it evolves and the fact that there's no vendor-provided repo is an issue ATM.

I've also looked into switching to balena-engine. Balena has the advantage of being tailored for IoT and low-memory and is stable (or doesn't evolve much depending on how you look at it). It's a fork of docker (moby) and is thus relatively close to the real thing albeit a bunch of missing features… which we are not depending on at the moment.

Works well for our use case but unfortunately, it's not enough for the Pi0-2W (or the Pi3A).

From what I observed, I believe this HW could support a moderate hotspot usage (limited number of apps, limited number of users) but the main blocker is the start-up phase: initializing the containers all at once create a spike in memory usage that forces for the engine and the system to kill process for OOM.

This could probably be worked around to some extent, and at the expense of maintenance. Some example:

• force an order on the compose
• individualize the services (instead of compose) to ensure all of them are started one by one
• reduce the base-system memory usage: removing services we dont need
• rewrite our scripts in bash or even sh
• use lighter base docker images for all services (not sure we can do more here)
• remove some containers based on target device and/or other conditions: hwclock could be spared on system with read-only packages
• better configure/optimize services inside our containers
• bundle some services together (kind of an anti-pattern): have the reverse-proxy serve homepage instead of proxying home service.

All of those require significant efforts for which gain is unknown upfront.

For this reason, I suggest we keep this ticket open but:

• Don't support 512M RAM devices (Pi0-2W, Pi3A) in v1
• Use balena as OCI engine. Forces us to stick to their features support. Switching back to docker is a breeze
• Discuss a strategy and open more focused tickets for next version

[kelson42]

Don't support 512M RAM devices (Pi0-2W, Pi3A) in v1

This is the path to follow IMHO.