Modified Docker services and attached CC-Slurm-Adapter

README.md

@@ -7,20 +7,19 @@ InfluxDB, LDAP, SLURM), or easily added by manual configuration (MariaDB).
 
 It includes the following containers:
 
-|Service full name|docker service name|port|
-| --- | --- | --- |
-|Slurm Controller service|slurmctld|6818|
-|Slurm Database service|slurmdbd|6817|
-|Slurm Rest service with JWT authentication|slurmrestd|6820|
-|Slurm Worker|node01|6818|
-|MariaDB service|mariadb|3306|
-|InfluxDB serice|influxdb|8086|
-|NATS service|nats|4222, 6222, 8222|
-|cc-metric-store service|cc-metric-store|8084|
-|OpenLDAP|openldap|389, 636|
+| Service full name | docker service name | port             |
+| ----------------- | ------------------- | ---------------- |
+| Slurm Controller  | slurmctld           | 6818             |
+| Slurm Database    | slurmdbd            | 6817             |
+| Slurm Rest (JWT)  | slurmrestd          | 6820             |
+| Slurm Worker      | node01              | 6818             |
+| NATS service      | nats                | 4222, 6222, 8222 |
+| cc-metric-store   | cc-metric-store     | 8084             |
+| KeyCloak          | keycloak            | 8080             |
+| OpenLDAP          | openldap            | 389, 636         |
 
 The setup comes with fixture data for a Job archive, cc-metric-store
-checkpoints, InfluxDB, MariaDB, and a LDAP user directory.
+checkpoints, and a LDAP user directory.
 
 ## Prerequisites
 
@@ -41,7 +40,7 @@ You can use:
 
 ```
 sudo groupadd docker
-sudo usermod -aG docker $USER 
+sudo usermod -aG docker $USER
 
 # restart after adding your docker with your user to sudo group
 sudo shutdown -r -t 0
@@ -50,59 +49,58 @@ sudo shutdown -r -t 0
 Note: You can install all these dependencies via predefined installation steps
 in `prerequisite_installation_script.sh`.
 
-If you are using different linux flavors, you will have to adapt
+If you are using different Linux flavors, you will have to adapt
 `prerequisite_installation_script.sh` as well as `setupDev.sh`.
 
-## Setup
+## Setup Procedure
 
 1. Clone `cc-backend` repository in chosen base folder: `$> git clone https://github.com/ClusterCockpit/cc-backend.git`
 
-2. Run `$ ./setupDev.sh`:  **NOTICE** The script will download files of a total
-   size of 338MB (mostly for the cc-metric-store data).
+2. Run the setup bash file: `$> ./setupDev.sh`: **NOTICE** The script will
+   download files of a total size of 338MB (mostly for the cc-metric-store
+   data).
 
 3. The setup-script launches the supporting container stack in the background
    automatically if everything went well. Run
 
-``` bash
+```bash
 ./cc-backend/cc-backend -server -dev
 ```
 
 to start `cc-backend`.
 
-4. By default, you can access `cc-backend` in your browser at
+1. By default, you can access `cc-backend` in your browser at
    `http://localhost:8080`. You can shut down the cc-backend server by pressing
-`CTRL-C`, remember to also shut down all containers via `$> docker-compose down`
-afterwards.
+   `CTRL-C`, remember to also shut down all containers via `$> docker-compose down`
+   afterwards.
 
-5. You can restart the containers with: `$> docker-compose up -d`.
+2. You can restart the containers with: `$> docker-compose up -d`.
 
 ## Credentials for logging into clustercockpit
 
 Credentials for the preconfigured demo user are:
 
-* User: `demo`
-* Password: `demo`
+- User: `demo`
+- Password: `demo`
 
 Credentials for the preconfigured LDAP user are:
 
-* User: `ldapuser`
-* Password: `ldapuser`
+- User: `ldapuser`
+- Password: `ldapuser`
 
 You can also login as regular user using any credential in the LDAP user
 directory at `./data/ldap/users.ldif`.
 
 ## Preconfigured setup between docker services and ClusterCockpit components
 
-When you are done cloning the cc-backend repo and once you execute `setupDev.sh`
-file, it will copy a preconfigured `config.json` from `misc/config.json` and
-replace the `cc-backend/config.json`, which will be used by cc-backend, once you
-start the server. The preconfigured config.json attaches to:
+When you are done cloning the cc-backend repo and once you execute `setupDev.sh` file, it will copy a preconfigured `config.json` from `misc/config.json` and replace the `cc-backend/config.json`, which will be used by cc-backend, once you start the server.
+The preconfigured config.json attaches to:
 
-### 1. MariaDB docker service on port 3306 (database: ccbackend)
+#### 1. OpenLDAP docker service on port 389
 
-### 2. OpenLDAP docker service on port 389
+#### 2. cc-metric-store docker service on port 8084
 
-### 3. cc-metric-store docker service on port 8084
+#### 3. cc-slurm-adapter is running on slurmctld docker service
 
 cc-metric-store also has a preconfigured `config.json` in
 `cc-metric-store/config.json` which attaches to NATS docker service on port 4222
@@ -128,7 +126,7 @@ docker services, you have to run **`bash`** command in those running services.
 > **`Example`**: You want to run slurm commands like `sinfo` or `squeue` or
 > `scontrol` on slurm controller, you cannot directly access it.
 
-You need to **`bash`** into the running service by using the following command:
+You need to open a **`bash`** session in the running service by using the following command:
 
 ```
 $ docker exec -it <docker service name> bash
@@ -137,7 +135,7 @@ $ docker exec -it <docker service name> bash
 $ docker exec -it slurmctld bash
 
 #or
-$ docker exec -it mariadb bash
+$ docker exec -it cc-metric-store bash
 ```
 
 Once you start a **`bash`** on any docker service, then you may execute any
@@ -158,26 +156,26 @@ cluster i.e. node01.
 In order to execute slurm commands, you may need to **`bash`** into the
 **`slurmctld`** docker service.
 
-``` bash
+```bash
 docker exec -it slurmctld bash
 ```
 
 Then you may be able to run slurm controller commands. A few examples without
 output are:
 
-``` bash
+```bash
 sinfo
 ```
 
 or
 
-``` bash
+```bash
 squeue
 ```
 
 or
 
-``` bash
+```bash
 scontrol show nodes
 ```
 
@@ -195,19 +193,9 @@ custom CURL commands.
 
 ## Known Issues
 
-* `docker-compose` installed on Ubuntu (18.04, 20.04) via `apt-get` can not
-correctly parse `docker-compose.yml` due to version differences. Install latest
-version of `docker-compose` from <https://docs.docker.com/compose/install/>
-instead.
-* You need to ensure that no other web server is running on ports 8080
-(cc-backend), 8082 (cc-metric-store), 8086 (InfluxDB), 4222 and 8222 (Nats), or
-3306 (MariaDB). If one or more ports are already in use, you have to adapt the
-related config accordingly.
-* Existing VPN connections sometimes cause problems with docker. If
-`docker-compose` does not start up correctly, try disabling any active VPN
-connection. Refer to
-<https://stackoverflow.com/questions/45692255/how-make-openvpn-work-with-docker>
-for further information.
+- `docker-compose` installed on Ubuntu (18.04, 20.04) via `apt-get` can not correctly parse `docker-compose.yml` due to version differences. Install latest version of `docker-compose` from <https://docs.docker.com/compose/install/> instead.
+- You need to ensure that no other web server is running on ports 8080 (cc-backend), 8084 (cc-metric-store), 4222 and 8222 (Nats). If one or more ports are already in use, you have to adapt the related config accordingly.
+- Existing VPN connections sometimes cause problems with docker. If `docker-compose` does not start up correctly, try disabling any active VPN connection. Refer to <https://stackoverflow.com/questions/45692255/how-make-openvpn-work-with-docker> for further information.
 
 ## Docker services and restarting the services
 
@@ -216,14 +204,14 @@ modify it.
 
 Whenever you modify it, please use
 
-``` bash
+```bash
 docker compose down
 ```
 
 in order to shut down all the services in all the VM’s (maininstance,
 nodeinstance, nodeinstance2) and then start all the services by using
 
-``` bash
+```bash
 docker compose up
 ```
 

dataGenerationScript.sh

@@ -1,4 +1,6 @@
 #!/bin/bash
+set -euo pipefail
+
 echo ""
 echo "|--------------------------------------------------------------------------------------|"
 echo "| This is Data generation script for docker services                                   |"
@@ -28,7 +30,7 @@ fi
 # This file automatically picked by mariadb after the docker service starts.
 if [ ! -d data/mariadb ]; then
     mkdir -p data/mariadb
-    cat > data/mariadb/01.databases.sql <<EOF
+    cat >data/mariadb/01.databases.sql <<EOF
 CREATE DATABASE IF NOT EXISTS \`ccbackend\`;
 EOF
 else
@@ -40,7 +42,7 @@ fi
 # This file automatically picked by openldap after the docker service starts.
 if [ ! -d data/ldap ]; then
     mkdir -p data/ldap
-    cat > data/ldap/add_users.ldif <<EOF
+    cat >data/ldap/add_users.ldif <<EOF
 dn: ou=users,dc=example,dc=com
 objectClass: organizationalUnit
 ou: users
@@ -65,11 +67,11 @@ fi
 # A simple configuration file for nats docker service.
 # Required because we need to execute custom commands after nats docker service starts.
 # This file automatically executed when the nats docker service starts.
-# After docker service starts, there is an infinite while loop that publises data for 'fritz' and 'alex' cluster 
+# After docker service starts, there is an infinite while loop that publises data for 'fritz' and 'alex' cluster
 # to subject 'hpc-nats' every 1 minute. Random data is generated only for node level metrics, not hardware level metrics.
 if [ ! -d data/nats ]; then
     mkdir -p data/nats
-    cat > data/nats/docker-entrypoint.sh <<EOF
+    cat >data/nats/docker-entrypoint.sh <<EOF
 #!/bin/sh
 set -e
 
@@ -125,15 +127,27 @@ else
     echo "'data/nats' already exists!"
 fi
 
-# prepare folders for influxdb3
-if [ ! -d data/influxdb ]; then
-    mkdir -p data/influxdb/data
-    mkdir -p data/influxdb/config
+if [ ! -d data/slurm/home/worker/CCSA ]; then
+    mkdir -p data/slurm/home/worker/CCSA
+
+    cat >data/slurm/home/worker/CCSA/config.json <<EOF
+{
+    "pidFilePath": "/home/worker/CCSA/daemon.pid",
+    "ipcSockPath": "/home/worker/CCSA/daemon.sock",
+    "lastRunPath": "/home/worker/CCSA/last_run",
+    "slurmPollInterval": 10,
+    "ccRestUrl": "http://host.docker.internal:8080",
+    "ccRestJwt": ""
+}
+EOF
+
+chmod 777 data/slurm/home/worker/CCSA/config.json
+
 else
-    echo "'data/influxdb' already exists!"
+    echo "'data/slurm/worker' already exists!"
 fi
 
 echo ""
 echo "|--------------------------------------------------------------------------------------|"
 echo "| Finished generating relevant files for docker services in data/                      |"
-echo "|--------------------------------------------------------------------------------------|"
+echo "|--------------------------------------------------------------------------------------|"

docker-compose.yml

@@ -90,6 +90,8 @@ services:
       - ${DATADIR}/slurm/state:/var/lib/slurm/d
     ports:
       - "6817:6817"
+    extra_hosts:
+    - "host.docker.internal:host-gateway"
 
   slurmdbd:
     container_name: slurmdbd
@@ -110,8 +112,8 @@ services:
       - "6819:6819"
 
   node01:
-    container_name: node01
-    hostname: node01
+    container_name: f0101
+    hostname: f0101
     build:
       context: ./slurm/worker
     depends_on:
@@ -147,3 +149,4 @@ services:
       - /etc/localtime:/etc/localtime:ro
     ports:
       - "6820:6820"
+

misc/config.json

@@ -8,8 +8,9 @@
   "jwts": {
     "max-age": "2000h"
   },
-  "db-driver": "mysql",
-  "db": "root:root@tcp(0.0.0.0:3306)/ccbackend",
+  "apiAllowedIPs": [
+    "*"
+  ],
   "ldap": {
     "url": "ldap://0.0.0.0",
     "user_base": "ou=users,dc=example,dc=com",

scripts/checkModules.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+set -euo pipefail
+
 cd scripts
 
 # Check if required perl modules are installed