Skip to content

mount trusted CA bundle in runner pods #1258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ktdreyer wants to merge 1 commit into
base: main
Choose a base branch
from
+246 −0
Open

mount trusted CA bundle in runner pods #1258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions components/operator/internal/handlers/sessions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
intstr "k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/util/retry"
)

Expand Down Expand Up @@ -1332,6 +1333,9 @@ func handleAgenticSessionEvent(obj *unstructured.Unstructured) error {
}
}

// Mount trusted CA bundle if present in the session namespace (e.g. OpenShift CA injection)
applyTrustedCABundle(config.K8sClient, sessionNamespace, pod)

// NOTE: Google credentials are now fetched at runtime via backend API
// No longer mounting credentials.json as volume
// This ensures tokens are always fresh and automatically refreshed
Expand Down Expand Up @@ -2180,6 +2184,55 @@ func deleteAmbientLangfuseSecret(ctx context.Context, namespace string) error {
return nil
}

// applyTrustedCABundle mounts the cluster's injected CA bundle into the runner
// container so it trusts cluster-internal TLS certificates (e.g. on OpenShift).
// Clusters without the ConfigMap are silently unaffected.
func applyTrustedCABundle(k8sClient kubernetes.Interface, namespace string, pod *corev1.Pod) {
cm, err := k8sClient.CoreV1().ConfigMaps(namespace).Get(
context.TODO(), types.TrustedCABundleConfigMapName, v1.GetOptions{})
if errors.IsNotFound(err) {
return
}
if err != nil {
log.Printf("Warning: failed to check for %s ConfigMap in %s: %v",
types.TrustedCABundleConfigMapName, namespace, err)
return
}
if _, ok := cm.Data["ca-bundle.crt"]; !ok {
if _, ok := cm.BinaryData["ca-bundle.crt"]; !ok {
log.Printf("Warning: %s ConfigMap in %s is missing required key ca-bundle.crt; skipping mount",
types.TrustedCABundleConfigMapName, namespace)
return
}
}
pod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{
Name: "trusted-ca-bundle",
VolumeSource: corev1.VolumeSource{
ConfigMap: &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: types.TrustedCABundleConfigMapName,
},
},
},
})
for i := range pod.Spec.Containers {
if pod.Spec.Containers[i].Name == "ambient-code-runner" {
pod.Spec.Containers[i].VolumeMounts = append(
pod.Spec.Containers[i].VolumeMounts,
corev1.VolumeMount{
Name: "trusted-ca-bundle",
MountPath: "/etc/pki/tls/certs/ca-bundle.crt",
SubPath: "ca-bundle.crt",
ReadOnly: true,
},
)
log.Printf("Mounted %s ConfigMap to /etc/pki/tls/certs/ca-bundle.crt in runner container",
types.TrustedCABundleConfigMapName)
break
}
Comment on lines +2218 to +2232
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 8, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Mount the bundle into init-hydrate and state-sync too.

This only patches ambient-code-runner. init-hydrate lives in pod.Spec.InitContainers and is configured with repo/S3 inputs (Lines 901-946), and state-sync is configured for S3 (Lines 1230-1248). On private-CA clusters, those containers can still fail TLS before the main runner starts, so private repo hydration/state sync is still broken unless they get the same trust bundle when present.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@components/operator/internal/handlers/sessions.go` around lines 2199 - 2213,
The current patch only mounts the trusted CA bundle into the ambient-code-runner
container; update the code to also mount the same VolumeMount into init
containers and any other containers that need it (specifically target init
container named "init-hydrate" in pod.Spec.InitContainers and the container
named "state-sync" in pod.Spec.Containers) so they receive the trusted CA before
network operations; reuse the same corev1.VolumeMount (Name "trusted-ca-bundle",
MountPath "/etc/pki/tls/certs/ca-bundle.crt", SubPath "ca-bundle.crt", ReadOnly
true) and log using types.TrustedCABundleConfigMapName, ensuring you iterate
pod.Spec.InitContainers to patch "init-hydrate" and pod.Spec.Containers to patch
both "state-sync" and "ambient-code-runner" (avoid duplicate mounts if already
present).

Copy link
Copy Markdown
Contributor Author

@ktdreyer ktdreyer Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai how can those require the CA?

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ktdreyer Good challenge — it depends on the deployment:

  • init-hydrate's backend API call (BACKEND_API_URL) uses plain http:// — no TLS at all, so definitely no CA needed there.
  • S3 (s3Endpoint) — if it's internal MinIO over HTTP (the typical default here), no CA needed. But if an operator configures an internal HTTPS S3 endpoint backed by a corporate CA, init-hydrate and state-sync would fail TLS.
  • Git clones in init-hydrate — if repos are hosted on an internal Gitlab/Gitea with a private CA cert, cloning over HTTPS would fail without the bundle.

So the containers can need it, but only when:

  1. The S3 endpoint is HTTPS + corporate CA, or
  2. Git repos are on a private-CA HTTPS host

If your target environments use plain-HTTP MinIO and public-CA git hosts, the runner container is the only one making connections (to Anthropic/Claude APIs, etc.) that realistically need the bundle. In that case you can safely ignore my comment. Your call on whether the edge cases (HTTPS S3, private git) are in scope for this PR.

Copy link
Copy Markdown
Contributor Author

@ktdreyer ktdreyer Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neither of those conditions are true. We'll ignore this for now.

}
}

// LEGACY: getBackendAPIURL removed - AG-UI migration
// Workflow and repo changes now call runner's REST endpoints directly

Expand Down
189 changes: 189 additions & 0 deletions components/operator/internal/handlers/sessions_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package handlers

import (
"context"
"fmt"
"testing"

"ambient-code-operator/internal/config"
Expand All @@ -13,6 +14,7 @@ import (
"k8s.io/apimachinery/pkg/runtime"
k8stypes "k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes/fake"
clienttesting "k8s.io/client-go/testing"
)

// setupTestClient initializes a fake Kubernetes client for testing
Expand Down Expand Up @@ -568,6 +570,193 @@ func TestDeleteAmbientVertexSecret_NotFound(t *testing.T) {
}
}

// TestApplyTrustedCABundle_ConfigMapPresent verifies that applyTrustedCABundle adds the volume
// and VolumeMount when the trusted-ca-bundle ConfigMap exists in the session namespace.
func TestApplyTrustedCABundle_ConfigMapPresent(t *testing.T) {
cm := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: types.TrustedCABundleConfigMapName,
Namespace: "session-ns",
},
Data: map[string]string{
"ca-bundle.crt": "--- fake CA data ---",
},
}
setupTestClient(cm)

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 1 {
t.Fatalf("expected 1 volume, got %d", len(pod.Spec.Volumes))
}
vol := pod.Spec.Volumes[0]
if vol.Name != "trusted-ca-bundle" {
t.Errorf("expected volume name 'trusted-ca-bundle', got %q", vol.Name)
}
if vol.ConfigMap == nil || vol.ConfigMap.Name != types.TrustedCABundleConfigMapName {
t.Errorf("expected ConfigMap volume sourced from %q", types.TrustedCABundleConfigMapName)
}

mounts := pod.Spec.Containers[0].VolumeMounts
if len(mounts) != 1 {
t.Fatalf("expected 1 VolumeMount, got %d", len(mounts))
}
m := mounts[0]
if m.Name != "trusted-ca-bundle" {
t.Errorf("expected mount name 'trusted-ca-bundle', got %q", m.Name)
}
if m.MountPath != "/etc/pki/tls/certs/ca-bundle.crt" {
t.Errorf("unexpected MountPath: %q", m.MountPath)
}
if m.SubPath != "ca-bundle.crt" {
t.Errorf("expected SubPath 'ca-bundle.crt', got %q", m.SubPath)
}
if !m.ReadOnly {
t.Error("expected ReadOnly=true")
}
}

// TestApplyTrustedCABundle_ConfigMapAbsent verifies that applyTrustedCABundle leaves the pod
// unchanged when the trusted-ca-bundle ConfigMap is not present in the session namespace.
func TestApplyTrustedCABundle_ConfigMapAbsent(t *testing.T) {
setupTestClient() // no ConfigMap

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 0 {
t.Errorf("expected no volumes, got %d", len(pod.Spec.Volumes))
}
if len(pod.Spec.Containers[0].VolumeMounts) != 0 {
t.Errorf("expected no VolumeMounts, got %d", len(pod.Spec.Containers[0].VolumeMounts))
}
}

// TestApplyTrustedCABundle_ExistingMountsPreserved verifies that applyTrustedCABundle appends
// to, rather than replacing, existing VolumeMounts on the runner container.
func TestApplyTrustedCABundle_ExistingMountsPreserved(t *testing.T) {
cm := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: types.TrustedCABundleConfigMapName,
Namespace: "session-ns",
},
Data: map[string]string{
"ca-bundle.crt": "--- fake CA data ---",
},
}
setupTestClient(cm)

existingMount := corev1.VolumeMount{
Name: "runner-token",
MountPath: "/var/run/secrets/ambient",
ReadOnly: true,
}
pod := &corev1.Pod{
Spec: corev1.PodSpec{
Volumes: []corev1.Volume{
{Name: "runner-token"},
},
Containers: []corev1.Container{
{
Name: "ambient-code-runner",
VolumeMounts: []corev1.VolumeMount{existingMount},
},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 2 {
t.Fatalf("expected 2 volumes (runner-token + trusted-ca-bundle), got %d", len(pod.Spec.Volumes))
}
mounts := pod.Spec.Containers[0].VolumeMounts
if len(mounts) != 2 {
t.Fatalf("expected 2 VolumeMounts, got %d", len(mounts))
}
// Existing mount must still be at index 0
if mounts[0].Name != "runner-token" {
t.Errorf("expected first mount to be 'runner-token', got %q", mounts[0].Name)
}
if mounts[1].Name != "trusted-ca-bundle" {
t.Errorf("expected second mount to be 'trusted-ca-bundle', got %q", mounts[1].Name)
}
}

// TestApplyTrustedCABundle_MissingKey verifies that applyTrustedCABundle leaves the pod
// unchanged when the ConfigMap exists but lacks the ca-bundle.crt key.
func TestApplyTrustedCABundle_MissingKey(t *testing.T) {
cm := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: types.TrustedCABundleConfigMapName,
Namespace: "session-ns",
},
Data: map[string]string{
"wrong-key.pem": "--- fake CA data ---",
},
}
setupTestClient(cm)

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 0 {
t.Errorf("expected no volumes when key is missing, got %d", len(pod.Spec.Volumes))
}
if len(pod.Spec.Containers[0].VolumeMounts) != 0 {
t.Errorf("expected no VolumeMounts when key is missing, got %d", len(pod.Spec.Containers[0].VolumeMounts))
}
}

// TestApplyTrustedCABundle_APIError verifies that applyTrustedCABundle leaves the pod
// unchanged when the ConfigMap GET returns a non-NotFound error.
func TestApplyTrustedCABundle_APIError(t *testing.T) {
fakeClient := fake.NewSimpleClientset()
fakeClient.PrependReactor("get", "configmaps", func(action clienttesting.Action) (bool, runtime.Object, error) {
return true, nil, fmt.Errorf("connection refused")
})
config.K8sClient = fakeClient

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 0 {
t.Errorf("expected no volumes on API error, got %d", len(pod.Spec.Volumes))
}
if len(pod.Spec.Containers[0].VolumeMounts) != 0 {
t.Errorf("expected no VolumeMounts on API error, got %d", len(pod.Spec.Containers[0].VolumeMounts))
}
}

// TestDeleteAmbientVertexSecret_NilAnnotations tests handling of secret with nil annotations
func TestDeleteAmbientVertexSecret_NilAnnotations(t *testing.T) {
secret := &corev1.Secret{
Expand Down
4 changes: 4 additions & 0 deletions components/operator/internal/types/resources.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ const (
// AmbientVertexSecretName is the name of the secret containing Vertex AI credentials
AmbientVertexSecretName = "ambient-vertex"

// TrustedCABundleConfigMapName is the CA bundle ConfigMap injected by OpenShift or provisioned
// manually on private-CA clusters. See issue #1247.
TrustedCABundleConfigMapName = "trusted-ca-bundle"

// CopiedFromAnnotation is the annotation key used to track secrets copied by the operator
CopiedFromAnnotation = "vteam.ambient-code/copied-from"
)
Expand Down
Loading