Doesn't work with recent systemd and/or docker releases

[mathstuf]

Something went wrong after an update and I haven't dug into it too much since this project seems dead. Basically, systemd sees the systemd-docker executable as dying with a status code of 1, but the container is still running. I don't know if there's some mixup with Docker communication going on or if systemd changed somewhere to break it. I suspect Docker, but I have no evidence.

The best solution seems to be migrating to rkt for running the images, but that is blocked by rkt/rkt#2392 without the ability to push images to a registry.

[dashesy]

I tried this and some other tricks, and each breaks in an interesting way. The right solution is rkt but that is not very popular unfortunately, specially in my case I needed docker volumes because of NVidia.
At the end I ended up running 2 services,

1. first one a docker run ... service

[Unit]
Description=Main service to run
After=docker.service
Wants=docker.service

[Service]
StandardOutput=journal
StandardError=journal
Restart=on-failure
RestartSec=10
ExecStartPre=-/usr/bin/docker stop Main
ExecStartPre=-/usr/bin/docker rm Main
ExecStart=/usr/bin/docker run -h Main -i -a stdout -a stderr --rm --name Main theimagenamehere bash
ExecStop=-/usr/bin/docker stop Main
ExecStopPost=-/usr/bin/docker rm Main

[Install]
WantedBy=multi-user.target

2. another service that will depend on the main service and actually runs my service:

[Unit]
Description=MyService
After=Main.service
Wants=Main.service

[Service]
StandardOutput=journal
StandardError=journal
Restart=on-failure
RestartSec=10
ExecStart=/opt/bin/docker_exec.sh Main /path/to/service --service-params

[Install]
WantedBy=multi-user.target

...

Here is docker_exec.sh:

#!/bin/bash

function docker_cleanup {
    docker exec $IMAGE bash -c "if [ -f $PIDFILE ]; then kill -TERM -\$(cat $PIDFILE); rm $PIDFILE; fi"
}

IMAGE=$1
PIDFILE=/tmp/docker-exec-$$
shift
trap 'kill $PID; docker_cleanup $IMAGE $PIDFILE' TERM INT
docker exec $IMAGE bash -c "echo \"\$\$\" > $PIDFILE; exec $*" &
PID=$!
wait $PID
trap - TERM INT
wait $PID

This way I can also exec multiple services in the same container.

[agend07]

which docker and systemd version makes it break?

[mathstuf]

Fedora 24 had 1.10.3 which worked. I'm now on 1.13.1 which isn't. systemd went from 229 to 233.

[filippog]

Broken on docker 17.12 and Debian stretch, fails with this error:

json: cannot unmarshal object into Go value of type string

Updating the vendored docker client fixes it.

[Halfwalker]

Same failure on Ubuntu 16.04.3
Docker is Server Version: 17.12.0-ce, specifically 17.12.0ce-0ub from the PPA
Systemd is 229-4ubuntu21

What did you mean "Updating the vendored docker client" ??

[mathstuf]

@Halfwalker I'm guessing it's these projects. Does Go not have a better solution than "embed source code" for dependency management?

[agend07]

@Halfwalker here is what worked for me - I've built the executable in fresh golang container and then checked the binary on Ubuntu 16.04.3 LTS

go get github.com/agend07/systemd-docker
cd /go/src/github.com/agend07/systemd-docker
./build
and systemd-docker binary is in bin folder

[Halfwalker]

I'm wondering how much it's really needed ... Plain old docker run --rm .... in the systemd unit file seems to be working fine. I can systemctl start|stop my_container and it all seems to work OK.

[mathstuf]

@Halfwalker What Type= is your Service? Systemd isn't doing any lifecycle management without this (e.g., if the container dies, it can't enforce the Restart= or RestartSec= actions)

[Halfwalker]

I used Plex as a test container - figured that would be a good stressor ...

[Unit]
Description=Plex Media Server
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=120
ExecStartPre=/usr/bin/docker pull plexinc/pms-docker
ExecStart=/usr/bin/docker run --rm --name=plex
--network=host
-e TZ=America/New_York
-e PLEX_UID=1000
-e PLEX_GID=1000
-v /stuff/Plex/config:/config
-v /stuff/Plex/transcode:/transcode
plexinc/pms-docker

ExecStop=/usr/bin/docker stop plex
ExecStopPost=/usr/bin/docker rm -f plex
ExecReload=/usr/bin/docker restart plex

Restart=always
RestartSec=20s
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

All the testing was done in a virtualbox VM. Regular systemctl start/stop plex worked fine. Rebooting the box worked fine.

[agend07]

I dont think it would work fine if the plex container crashed - as long as docker service would keep working systemd wouldn't know anything wrong happend cause it would monitor docker service, not your plex container. It is the part that when u start docker container u talk to docker, and docker starts another process (with plex) - which confuses systemd.

So check if systemd would restart your plex container after u kill it with 'kill pid' or 'docker kill'

[Halfwalker]

Right - then systemd is monitoring docker, not the plex process in docker. With the unit file above though, docker would do the restart of the plex container. So while systemd wouldn't know that the plex container took a hit and restarted, the end result is the same : the plex container was restarted. systemd would only step in if docker itself died.

My preference would be for systemd to know about plex though, via systemd-docker. That just seems much cleaner.

A new build of systemd-docker seems to work. I've been pulling systemd-docker in via ansible for installs, but had to switch to plain docker run when it started failing. Now looking at what's needed to actually build a "latest" systemd-docker on a target system via ansible. @agend07 are you going to do a new release to handle the latest docker ?

[Halfwalker]

Here's a simple way to build a new version of systemd-docker if you don't want to install golang etc. Requires docker though :)

docker run --rm -it -v "/usr/local/bin":/output golang:1.9 /bin/bash -c "go get github.com/agend07/systemd-docker && cd /go/src/github.com/agend07/systemd-docker && ./build && cp bin/systemd-docker /output"

[firex2]

i just updated from 16.04.x to 18.04.1 and now my docker-systemd is broken (error: json: cannot unmarshal object into Go value of type string)

unfortunately i cant test by building my own version because "go get github.com/agend07/systemd-docker" gives my an error: fatal: repository 'https://github.com/weaveworks/docker/' not found

what can i do to get it working again?

[agend07]

@firex2 try to build it again, I forked some repo with mflag into mine github. It's building but I'm not sure it would work on 18.04.1