Skip to content

Commit f67074c

Browse files
Michal Tichákknopers8
authored andcommitted
[OCTRL-1052] Provide description how to run readout with CRU from ECS
1 parent ddc3b5b commit f67074c

File tree

1 file changed

+44
-30
lines changed

1 file changed

+44
-30
lines changed

docs/running_docker.md

Lines changed: 44 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
> This method is **not intended for production use**.
55
> It serves only as a **proof of concept** for testing Docker images as part of an existing pipeline.
66
>
7-
> Currently, it has been tested with the `alma9-flp-node` image running the *readout* component.
7+
> Currently, it was tested with the `alma9-flp-node` image running the *readout* component with CRU.
88
99
---
1010

@@ -45,55 +45,69 @@ For example, to run readout, modify the `_plain_command` section of [`readout.ya
4545
When running readout, we successfully used the following command inside the `alma9-flp-node` image:
4646

4747
```bash
48-
sudo /usr/bin/docker run --name readout --replace \
49-
--user "$(id -u flp):$(id -u flp)" \
48+
sudo -E docker run --name readout --replace \
49+
--user flp -v /etc/group:/etc/group:ro -v /etc/passwd:/etc/passwd:ro \
50+
--privileged \
5051
--network=host --ipc=host \
51-
-e O2_DETECTOR -e O2_PARTITION -e OCC_CONTROL_PORT \
52-
-e O2_SYSTEM -e O2_ROLE \
52+
-v /tmp:/tmp \
53+
-v /lib/modules/$(uname -r):/lib/modules/$(uname -r) \
54+
-e O2_DETECTOR -e O2_PARTITION -e OCC_CONTROL_PORT -e O2_SYSTEM -e O2_ROLE \
5355
gitlab-registry.cern.ch/aliceo2group/dockerfiles/alma9-flp-node:2 \
5456
/opt/o2/bin/o2-readout-exe
5557
```
5658

5759
> 🧩 **Note**
5860
> We are not claiming that this is the most efficient way how to run this image, just that it works.
61+
>
62+
#### Explanation of command
63+
64+
Let's explain all parts of the `docker run` command and what is their purpose. As RHEL uses Podman by default
65+
instead of docker we are going to comment on Podman parameters, but docker should be equivalent or pretty close.
66+
67+
- `sudo -E` switch to the rootful mode of Podman and pass all of the environment variables
68+
- `docker run --name readout --replace` start container with the name readout and replace already existing container
69+
- `--privileged` runs container with extended privileges and allows access to the devices and other resources
70+
on host system with the same privileges as user running the container (`flp` in our case). Readout communicates
71+
directly with CRUs and others connected via PCIe so we need to make these available inside the container.
72+
- `--user flp -v /etc/group:/etc/group:ro -v /etc/passwd:/etc/passwd:ro` run container as flp user provide same user
73+
and group settings as on a host machine. This is necessary as readout is using shared memory which is mapped under
74+
flp user and needs access to [BAR](https://github.com/AliceO2Group/ReadoutCard?tab=readme-ov-file#bar-interface)
75+
interfaces of CRU which belong to the `pda` group. We used this way of setting up user privileges and ids as there is
76+
no guarantee that ids would match if we would hardwire user into docker image
77+
- `--network=host` bind container's network to the host's. This allows services running outside of docker to communicate
78+
with internals (eg. gRPC, IL, ... ). It should be changed to open just required ports.
79+
- `--ipc=host` readout uses FairMQ which communicates through shared memory Inter-Process Communication.
80+
As we are now running readout inside the docker with the rest of data distribution running bare metal we
81+
need to have access to the hosting OS shared memory. This is the main requirement for running as rootful.
82+
If we run container in rootless mode with this flag set, Podman switches into elevated privileges mode that blocks
83+
other cli commands from the same user command until this command is finished unless running under root.
84+
We tried to just bind `/dev/shm/` but readout failed on permission errors.
85+
- `-v /tmp:/tmp` binds `/tmp` of host to the container for monitoring purposes.
86+
- `-v /lib/modules/$(uname -r):/lib/modules/$(uname -r)` binds folder with host's kernel modules for the usage of PDA
87+
used by readout itself.
88+
- `e ...` pass environment variables required by readout to the container
5989

6090
#### Environment Variables
6191

62-
To identify all required environment variables:
92+
---
93+
94+
## Tips and Tricks
95+
96+
- To identify all required environment variables:
6397

6498
1. Open the **ECS GUI**.
6599
2. Go to the **Environment Details** page for the relevant task.
66100
3. Review the variables defined there — these match those used when running the binary outside Docker.
67101

68-
#### Shared Memory Communication
69-
70-
To enable shared memory communication between processes, add the `--ipc=host` flag when running the container.
71-
However, doing so requires **elevated privileges**.
72-
73-
While **Podman** can run without root privileges, it pauses other Podman processes for the same user.
74-
This means commands like `podman ps -a` or starting multiple containers in parallel will not work.
75-
76-
Therefore, you should run containers using the same user as the rest of the pipeline:
77-
78-
```bash
79-
--user "$(id -u flp):$(id -u flp)"
80-
```
81-
82-
This ensures shared memory segments are created under the same user context.
83-
84-
---
85-
86-
## Tips and Tricks
87-
88-
* Production systems running RHEL do not install native Docker via:
102+
- Production systems running RHEL do not install native Docker via:
89103

90104
```bash
91105
dnf install docker
92106
```
93107

94-
Instead, they use **Podman**, which emulates Docker’s behavior but may differ in certain aspects.
108+
Instead, they use Podman, which emulates Docker’s behavior but may differ in certain aspects.
95109

96-
* To check whether ECS has started a container, run:
110+
- To check whether ECS has started a container, run:
97111

98112
```bash
99113
docker ps -a
@@ -107,7 +121,7 @@ This ensures shared memory segments are created under the same user context.
107121
> su - flp
108122
> ```
109123
110-
* To view container logs directly from Docker:
124+
- To view container logs directly from Docker:
111125
112126
```bash
113127
docker logs <container-id|name>

0 commit comments

Comments
 (0)