[Discussing] Workaround to deploy on kubernetes with plugins

[pnck]

Reference issue: #978

SS pod cannot get started due to the mounting point at /usr/local/bin. K8s' mount mechanism works different from docker's, which would overwrite the underlying files instead of keeping them togetherr. (There is no overlay filesystem.)

So the solution is either mount plugin volumes into another path, or modify the docker image to ensure that ssserver at a place that won't get overwritten.

This draft shows a workaround to download and serve with plugins.

Besides the test-connection part is suggested removal because this pod will always be ready before the main pod, thus
fails invariably and making no sense.

[pnck]

@zonyitoo

let me introduce some basic conceptions.

k8s' workload unit is called a pod, which may contain multiple containers but typically only one.
since containers don't share their filesystem, multi-container pods are used for, e.g. a website including frontend and backend servers. fe server and be server (container image) are built independently but organized into a single pod so that when at the site bootstrap, both server start synchonously.

in this ss server case instead, we want the plugins had been injected into the ssserver container before it's serving. there is no need (and should not) to make a multi-container pod, because that will isolate plugins from the main ssserver program.
so the proper way is, to setup an init container, which prepares (download) plugins before the ssserver container schedules, and save the binaries in a temporary volume which belongs to the pod that can be mount to the next (the main) container. since the pod itself doesn't destroy, the temp volume is still there.

the problem we are facing is, the original specified mounting path /usr/local/bin contains the ssserver and entrypoint.sh executables. when the main container starts, this files will be replaced by the mounted files, because k8s isn't providing an overlay filesystem, which differ from docker. and the container will fail to start because there is no entry files at all.

so here is the simplest solution that changes the mounting path to /usr/local/sbin, which let the ssserver container start normally while having plugins in another path in $PATH. then the plugin options shall work.

 

but in fact, the k8s deployment is still far from available. the ss container only serves connections from inside the cluster (or on the node machine) at the moment. i'm still doing experiments on it.

[zonyitoo]

Make sense.