Fdp update edpm #3467

mnietoji · 2025-11-07T14:44:36Z

[multiple] Add fdp_update_edpm role for EDPM node updates

[fdp_update_edpm] Implement EDPM node update automation for FDP updates:

Role fdp_update_edpm: Updates EDPM nodes declaratively via Kubernetes CRs
- Patches OpenStackDataPlaneNodeSet CRs with updated container images
- Configures package updates via edpm_bootstrap_packages
- Sets up registry authentication and CA certificates
- Creates OpenStackDataPlaneDeployment to apply changes
- Includes hypervisor firewall configuration for registry access
Fix hypervisor firewall configuration
- Add delegate_to to execute iptables on correct hypervisor host
- Previously executed on localhost instead of hypervisor
Integration in post-deployment.yml after control plane updates
Zuul CI configuration for automated testing

[fdp_update_container_images] Fix to properly update OpenStackVersion CR

Add set_fact task to build customContainerImages dict correctly

Assisted-By: Claude noreply@anthropic.com
Signed-off-by: Miguel Angel Nieto Jimenez mnietoji@redhat.com

michburk

Looks like this pr isn't split properly and also contains the changes from #3466

softwarefactory-project-zuul · 2025-11-07T18:33:49Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/381a8462756e461fa40fdad368f58ea3

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 44m 12s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 14s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 30m 53s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 27s
✔️ noop SUCCESS in 0s
❌ cifmw-pod-ansible-test TIMED_OUT in 30m 56s
❌ cifmw-pod-pre-commit FAILURE in 7m 51s
✔️ build-push-container-cifmw-client SUCCESS in 18m 58s
✔️ cifmw-molecule-fdp_update_container_images SUCCESS in 2m 18s
✔️ cifmw-molecule-fdp_update_edpm SUCCESS in 2m 22s

softwarefactory-project-zuul · 2025-11-07T21:35:12Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8dae75330b79473b922c4f394be22509

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 38m 24s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 49s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 24m 29s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 38s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 10s
❌ cifmw-pod-pre-commit FAILURE in 7m 49s
✔️ build-push-container-cifmw-client SUCCESS in 19m 15s
✔️ cifmw-molecule-fdp_update_container_images SUCCESS in 2m 19s
✔️ cifmw-molecule-fdp_update_edpm SUCCESS in 2m 17s

mnietoji · 2025-11-07T23:40:57Z

Looks like this pr isn't split properly and also contains the changes from #3466

This pr depends on the other one. When the other one is merged, i will rebase this one and I will only have one commit here

softwarefactory-project-zuul · 2025-11-08T01:37:02Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/a31750715d0943f7889539e20267c60d

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 42m 49s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 17m 33s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 26m 30s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 36s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 52s
❌ cifmw-pod-pre-commit FAILURE in 7m 58s
✔️ build-push-container-cifmw-client SUCCESS in 22m 17s
✔️ cifmw-molecule-fdp_update_container_images SUCCESS in 1m 44s
✔️ cifmw-molecule-fdp_update_edpm SUCCESS in 1m 52s

danpawlik · 2025-11-12T15:35:21Z

roles/fdp_update_container_images/defaults/main.yml

+# ============================================================================
+
+# Base directory for artifacts and temporary files
+cifmw_fdp_update_container_images_basedir: "{{ cifmw_basedir | default(ansible_user_dir ~ '/ci-framework-data') }}"


remove default. It should start using group_vars/all.yml

fdp_update_container_images not in this PR, this was managed in a different PR

danpawlik · 2025-11-12T15:36:29Z

roles/fdp_update_container_images/defaults/main.yml

+
+# Repository configuration
+cifmw_fdp_update_container_images_repo_name: "custom-repo"
+cifmw_fdp_update_container_images_repo_baseurl: ""  # REQUIRED - must be set by user


just a nit: do we really want to have so much parameters available?

fdp_update_container_images not in this PR, this was managed in a different PR

danpawlik · 2025-11-12T15:37:42Z

roles/fdp_update_edpm/tasks/configure_ca_cert.yml

@@ -0,0 +1,65 @@
+---


are you sure we need that task?

yes, we want to download images from openshift registry. That registry was not used before by the compute node. We need to either configure it as a insecure registry or configure the certificate in the compute node.

Are you sure? Current CI job workflow seems not "catching" into your exception. Don't think it is needed, but if yes, would be surprise and probably you will not use ci-framework tool for preparing CI jobs.

Edpm nodes are downloading modified images from openshift registry. It is needed to configure certificate or configure the registry as insecure.
This is needed because images modified are stored in openshift registry. For regular deployment, images are downloaded from external registries and it is configured the certificates needed to access those registries or it is configured as insecure.

danpawlik · 2025-11-12T15:38:36Z

roles/fdp_update_edpm/tasks/create_deployment.yml

+        _cifmw_fdp_update_edpm_deployment_result.resources | length > 0 and
+        _cifmw_fdp_update_edpm_deployment_result.resources[0].status.conditions |
+        selectattr('type', 'equalto', 'Ready') |
+        map(attribute='status') | first | default('False') == 'True'


default is needed here?

It is there in case the filter first return empty, so It set it to False

danpawlik · 2025-11-12T15:39:43Z

roles/fdp_update_edpm/tasks/setup_hypervisor_firewall.yml

@@ -0,0 +1,59 @@
+---


not sure we need that task.

yes, it is needed because our compute nodes does not have connectivity to the openshift registry in which we store the updated images. I only add a route so that compute nodes can access registry

yes, i commented before the reason
0c3f13a#r2592543239

danpawlik · 2025-12-05T13:01:42Z

roles/fdp_update_edpm/tasks/process_nodeset.yml

+      register: _cifmw_fdp_update_edpm_current_logins_result
+      failed_when: false
+
+    - name: Parse existing registry logins


this fact and later using this fact is just: xDˣᴰ

Updated in a way it is easier to understand

danpawlik

Please take a look what AI generates you back.
Also some part of the code was already done in fdp_update_container_images.

evallesp · 2025-12-05T12:58:04Z

roles/fdp_update_container_images/tasks/process_image.yml

    state: build

+- name: Build custom container images dict
+  ansible.builtin.set_fact:


I agree, I'm unsure why this change is required.

evallesp · 2025-12-05T13:12:00Z

roles/fdp_update_edpm/tasks/main.yml

+  when:
+    - cifmw_fdp_update_edpm_auto_deploy | bool
+    - not cifmw_fdp_update_edpm_dry_run | bool
+    - _cifmw_fdp_update_edpm_updated_nodesets | default([]) | length > 0


indeed not required. We've already added the default to the default role. And it'd not be te way to import this task instead importing this role.

evallesp · 2025-12-05T13:13:41Z

roles/fdp_update_edpm/tasks/process_nodeset.yml

+- name: Patch container images in nodeset
+  when:
+    - cifmw_fdp_update_edpm_containers_enabled | bool
+    - _cifmw_fdp_update_edpm_updated_images is defined


indeed is not necessary, as this is defined in the defualts of the role, and it doesn't seems that this task is included outside the natural include_role process. So then this var is always defined.

roles/fdp_update_edpm/tasks/process_nodeset.yml

roles/fdp_update_edpm/tasks/update_host_packages.yml

evallesp · 2025-12-05T13:34:20Z

roles/fdp_update_edpm/tasks/update_host_packages.yml

+
+- name: Add new repo to repos list
+  ansible.builtin.set_fact:
+    _cifmw_fdp_update_edpm_merged_repos: "{{ _cifmw_fdp_update_edpm_bootstrap_repos_list + [_cifmw_fdp_update_edpm_new_repo] }}"


(non-blocking) question: is required this to be a fact? I don't see it used anywhere outside this file.

It is used several times in the file

(non-blocking) suggestion: let's move into a variable.

As it is a non-blocking issue, i can check these kind of improvementes later on in a different PR

roles/fdp_update_edpm/tasks/update_container_images.yml

mnietoji

Checked most of the comments, i will continue tomorrow

mnietoji · 2025-12-10T14:44:38Z

roles/fdp_update_container_images/tasks/process_image.yml

    state: build

+- name: Build custom container images dict
+  ansible.builtin.set_fact:


This is what is happening with current code without set_fact. I added a debug task before to show the content of the variable.

TASK [fdp_update_container_images : Debug - Show new image path internal msg=New image path (internal): {{ _cifmw_fdp_update_container_images_new_image_path_internal }}] ***
Wednesday 10 December 2025 14:38:01 +0000 (0:00:01.774) 0:00:26.053 ****
Wednesday 10 December 2025 14:38:01 +0000 (0:00:01.774) 0:00:26.052 ****
ok: [localhost] =>
msg: 'New image path (internal): image-registry.openshift-image-registry.svc:5000/openstack/fdp-update-ceilometersgcoreimage-1765377456'

TASK [fdp_update_container_images : Patch OpenStackVersion CR state=patched, api_version=core.openstack.org/v1beta1, kind=OpenStackVersion, name={{ cifmw_fdp_update_container_images_openstack_cr_name }}, namespace={{ cifmw_fdp_update_container_images_namespace }}, definition={'spec': {'customContainerImages': {'{{ image_entry.key }}': '{{ _cifmw_fdp_update_container_images_new_image_path_internal }}'}}}] ***
Wednesday 10 December 2025 14:38:01 +0000 (0:00:00.037) 0:00:26.091 ****
Wednesday 10 December 2025 14:38:01 +0000 (0:00:00.037) 0:00:26.089 ****
[WARNING]: unknown field "spec.customContainerImages.{{ image_entry.key }}"
ok: [localhost]

The problem is that Ansible doesn't evaluate {{ image_entry.key }} when used directly as a YAML dictionary key. It sends the literal string "{{ image_entry.key }}" to Kubernetes instead of the actual value like ceilometerSgcoreImage. The solution is to build the dictionary as a string template first, then convert it with from_yaml filter - this forces proper variable evaluation before creating the dictionary.

Instead of using the set_fact, i have changed to this other sintax:
customContainerImages: "{{ { image_entry.key: _cifmw_fdp_update_container_images_new_image_path_internal } }}"

mnietoji · 2025-12-10T15:19:32Z

roles/fdp_update_edpm/defaults/main.yml

+# Dry run - show changes without applying
+cifmw_fdp_update_edpm_dry_run: false
+
+# ============================================


fdp_update_container_images patches images with ovn in FDP repo and uploads them to the openshift registry. It also updates openstackversion to force pods using those images to be updated.
fdp_update_edpm uses images already created by fdp_update_container_images. The goal of this role is to update images in compute nodes (they are not pods, just containers). It also update ovs packages in compute host

mnietoji · 2025-12-10T15:28:00Z

roles/fdp_update_edpm/tasks/cleanup_hypervisor_firewall.yml

@@ -0,0 +1,62 @@
+---


This role is updating containers running in compute nodes. Images are stored in openshift registry. I could have chosen to use any external registry to store that images, but I liked the idea of not depending on any other external resources and I do not need to keep those images, so openshift registry is nice.
But compute nodes are not connected to openshift baremetal network. One solution may have been to add a new vlan in compute nodes, configure the switch and allow compute nodes to access baremetal network, but i didnt want to have a modification on the templates only because i need it for testing.
What I am doing is to add some firewall rules in hypervisor to access openshift baremetal network from compute nodes using controlplane network

mnietoji · 2025-12-10T15:38:14Z

roles/fdp_update_edpm/tasks/fetch_nodesets.yml

+
+- name: Fail if no NodeSets found
+  ansible.builtin.fail:
+    msg: "No OpenStackDataPlaneNodeSets found in namespace {{ cifmw_fdp_update_edpm_namespace }}"


yes, it make sense:
if cifmw_fdp_update_edpm_nodeset_name == 'all' executes: oc get osdpns
if cifmw_fdp_update_edpm_nodeset_name != 'all' executes: oc get osdpns cifmw_fdp_update_edpm_nodeset_name

In both cases, if there is no nodeset created, _cifmw_fdp_update_edpm_nodesets will be empty

mnietoji · 2025-12-10T15:39:15Z

roles/fdp_update_edpm/tasks/main.yml

+# ============================================
+
+- name: Setup hypervisor firewall for registry access
+  ansible.builtin.include_tasks: setup_hypervisor_firewall.yml


yes, i commented before the reason
0c3f13a#r2592543239

roles/fdp_update_edpm/tasks/update_host_packages.yml

mnietoji · 2025-12-10T16:59:18Z

roles/fdp_update_edpm/tasks/update_host_packages.yml

+
+- name: Add new repo to repos list
+  ansible.builtin.set_fact:
+    _cifmw_fdp_update_edpm_merged_repos: "{{ _cifmw_fdp_update_edpm_bootstrap_repos_list + [_cifmw_fdp_update_edpm_new_repo] }}"


It is used several times in the file

mnietoji · 2025-12-10T17:02:38Z

roles/fdp_update_edpm/tasks/validate.yml

+    that:
+      - cifmw_fdp_update_edpm_repo_baseurl is defined
+      - cifmw_fdp_update_edpm_repo_baseurl | length > 0
+      - cifmw_fdp_update_edpm_packages is defined


mnietoji · 2025-12-10T17:07:12Z

roles/fdp_update_edpm/tasks/validate.yml

+
+- name: Initialize updated nodesets tracking list
+  ansible.builtin.set_fact:
+    _cifmw_fdp_update_edpm_updated_nodesets: []


why?
it is created here empty and later on it is updated for each nodeset and and is used to decide if a deployment is created or not and also for the debug message at the end

softwarefactory-project-zuul · 2025-12-10T18:54:24Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/32e75138ecb8467586f36758556b4c4f

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 44m 55s
❌ podified-multinode-edpm-deployment-crc FAILURE in 36m 41s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 42m 19s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 1h 23m 36s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 36s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 41s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 11s
✔️ build-push-container-cifmw-client SUCCESS in 21m 56s
✔️ cifmw-molecule-fdp_update_container_images SUCCESS in 2m 19s
✔️ cifmw-molecule-fdp_update_edpm SUCCESS in 2m 24s

softwarefactory-project-zuul · 2025-12-11T14:50:24Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7bdf595fca3c4cf6adb0786fddcda0f5

❌ openstack-k8s-operators-content-provider FAILURE in 7m 11s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 08s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 06s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 59s
✔️ build-push-container-cifmw-client SUCCESS in 19m 47s
✔️ cifmw-molecule-fdp_update_container_images SUCCESS in 2m 26s
✔️ cifmw-molecule-fdp_update_edpm SUCCESS in 2m 26s

softwarefactory-project-zuul · 2025-12-11T16:44:16Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d03970aa66cb4aa2b09e0d3a4c74d2e4

❌ openstack-k8s-operators-content-provider FAILURE in 5m 23s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 31s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 30s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 01s
✔️ build-push-container-cifmw-client SUCCESS in 18m 07s
✔️ cifmw-molecule-fdp_update_container_images SUCCESS in 2m 17s
✔️ cifmw-molecule-fdp_update_edpm SUCCESS in 2m 20s

mnietoji · 2025-12-22T10:13:31Z

Please take a look what AI generates you back. Also some part of the code was already done in fdp_update_container_images.

Not same code, maybe name of the file is the same but it is doing it a different thing

[fdp_update_edpm] Implement EDPM node update automation for FDP updates: - Role fdp_update_edpm: Updates EDPM nodes declaratively via Kubernetes CRs * Patches OpenStackDataPlaneNodeSet CRs with updated container images * Configures package updates via edpm_bootstrap_packages * Sets up registry authentication and CA certificates * Creates OpenStackDataPlaneDeployment to apply changes * Includes hypervisor firewall configuration for registry access - Fix hypervisor firewall configuration * Add delegate_to to execute iptables on correct hypervisor host * Previously executed on localhost instead of hypervisor - Integration in post-deployment.yml after control plane updates - Zuul CI configuration for automated testing [fdp_update_container_images] Fix to properly update OpenStackVersion CR * Add set_fact task to build customContainerImages dict correctly Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Miguel Angel Nieto Jimenez <mnietoji@redhat.com>

mnietoji · 2025-12-22T14:18:50Z

Hi, have rebased the patch and gone through alll pending comments. Testing is green again

mnietoji requested a review from a team as a code owner November 7, 2025 14:44

mnietoji force-pushed the fdp_update_edpm branch from 8d17965 to 24774d6 Compare November 7, 2025 15:09

mnietoji closed this Nov 7, 2025

mnietoji reopened this Nov 7, 2025

mnietoji added the Ready For Review label Nov 7, 2025

mnietoji mentioned this pull request Nov 7, 2025

Add FDP update automation for EDPM deployments #3432

Closed

mnietoji force-pushed the fdp_update_edpm branch from 24774d6 to 728f0fb Compare November 7, 2025 16:31

github-actions bot removed the Ready For Review label Nov 7, 2025

mnietoji force-pushed the fdp_update_edpm branch 3 times, most recently from b645ded to 02be96f Compare November 7, 2025 16:48

michburk previously requested changes Nov 7, 2025

View reviewed changes

mnietoji force-pushed the fdp_update_edpm branch from 02be96f to c06f97a Compare November 7, 2025 19:55

mnietoji force-pushed the fdp_update_edpm branch 2 times, most recently from d4a6725 to 6410812 Compare November 7, 2025 23:39

mnietoji force-pushed the fdp_update_edpm branch 2 times, most recently from f09d329 to 2f1a170 Compare November 7, 2025 23:52

mnietoji force-pushed the fdp_update_edpm branch from 2f1a170 to 8e50296 Compare November 11, 2025 08:53

github-actions bot added the Ready For Review label Nov 11, 2025

danpawlik reviewed Nov 12, 2025

View reviewed changes

mnietoji force-pushed the fdp_update_edpm branch from 8e50296 to 8f40669 Compare November 13, 2025 15:15

github-actions bot removed the Ready For Review label Nov 13, 2025

danpawlik reviewed Dec 5, 2025

View reviewed changes

danpawlik requested changes Dec 5, 2025

View reviewed changes

evallesp reviewed Dec 5, 2025

View reviewed changes

github-actions bot added the Ready For Review label Dec 5, 2025

mnietoji commented Dec 10, 2025

View reviewed changes

mnietoji force-pushed the fdp_update_edpm branch from 0c3f13a to 422e6e7 Compare December 10, 2025 17:08

github-actions bot removed the Ready For Review label Dec 10, 2025

mnietoji force-pushed the fdp_update_edpm branch from 422e6e7 to a3cf0b2 Compare December 11, 2025 14:29

mnietoji force-pushed the fdp_update_edpm branch from a3cf0b2 to 276c995 Compare December 11, 2025 16:25

mnietoji force-pushed the fdp_update_edpm branch 2 times, most recently from b0fbe03 to e75e75f Compare December 12, 2025 08:31

github-actions bot added the Ready For Review label Dec 12, 2025

mnietoji added Ready For Review and removed Ready For Review labels Dec 12, 2025

mnietoji requested review from danpawlik, evallesp and michburk December 12, 2025 16:19

mnietoji closed this Dec 22, 2025

mnietoji reopened this Dec 22, 2025

github-actions bot removed the Ready For Review label Dec 22, 2025

mnietoji force-pushed the fdp_update_edpm branch from e75e75f to 39a340c Compare December 22, 2025 10:14

mnietoji force-pushed the fdp_update_edpm branch from 39a340c to 93e41a8 Compare December 22, 2025 10:24

github-actions bot added the Ready For Review label Dec 22, 2025

evallesp approved these changes Dec 22, 2025

View reviewed changes

Fdp update edpm #3467

Are you sure you want to change the base?

Fdp update edpm #3467

Conversation

mnietoji commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michburk left a comment

Choose a reason for hiding this comment

Uh oh!

softwarefactory-project-zuul bot commented Nov 7, 2025

Uh oh!

softwarefactory-project-zuul bot commented Nov 7, 2025

Uh oh!

mnietoji commented Nov 7, 2025

Uh oh!

softwarefactory-project-zuul bot commented Nov 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mnietoji Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danpawlik left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mnietoji left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mnietoji commented Nov 7, 2025 •

edited

Loading

mnietoji Dec 11, 2025 •

edited

Loading