Cloud Run work pool: Secret-based env vars ignored in v1; solution is Cloud Run v2 with valueSource.secretKeyRef

[zzstoatzz]

This discussion was created from a Slack thread conversation.

Original Thread: https://prefect-community.slack.com/archives/C04DZJC94DC/p1761065531562869

---

Summary
A user attempted to configure secret-based environment variables for a Prefect 3 Cloud Run work pool. Using a Cloud Run Job v1 (apiVersion run.googleapis.com/v1, kind: Job), they set container env in job_body and tried to pass secret references via job_variables. However, the resulting Cloud Run Job only contained Prefect-injected envs (PREFECT_API_URL, PREFECT__FLOW_RUN_ID, etc.); all custom secret-based envs were ignored or stringified. Switching the work pool to Cloud Run v2 resolved the issue.

Key findings

• Cloud Run v1 worker behavior:

  • The v1 worker overwrites containers[].env when building the Job request; hardcoding env in job_body is ignored.
  • The built-in variables define env as a dict[str, str]; Prefect coerces it to strings and does not pass through objects like valueFrom.secretKeyRef.
  • Attempting to set job_variables.env as a list results in Pydantic coercion errors (it expects a mapping), e.g., AttributeError: 'list' object has no attribute 'items'.
  • Outcome: Secret-based env cannot be reliably passed to the container via the v1 worker’s default schema. Prefect-injected envs still appear as expected.

• Cloud Run Job v1 vs v2 schema:

  • Job v1 uses Kubernetes-style secret references: env[].valueFrom.secretKeyRef with name/key (e.g., key: "latest").
  • Job v2 uses env[].valueSource.secretKeyRef with secret/version.
  • For Jobs v1, correct nesting is spec.template.template.spec.containers[].env (double "template").

• Successful approach: Cloud Run v2 worker

  • Create a Cloud Run v2 work pool and wire a dedicated container_env array (list of env objects) into job_body.template.template.containers[0].env.
  • Provide secret-based envs with valueSource.secretKeyRef.secret/version.
  • This allows secret manager references to pass through unchanged and be honored by Cloud Run.

Actionable steps

1. Create and export the v2 base job template

prefect work-pool create my-cr-v2 --type cloud-run-v2
prefect work-pool get-default-base-job-template --type cloud-run-v2 --file base_v2.json

2. Edit base_v2.json

• Add a new variables property container_env as an array of objects, each with name/value or name/valueSource.secretKeyRef.
• Map it into the container env:

job_body.template.template.containers[0].env = "{{ container_env }}"

Recommended variables schema patch excerpt

"container_env": {
  "title": "Container environment variables",
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "value": { "type": ["string", "null"] },
      "valueSource": {
        "type": "object",
        "properties": {
          "secretKeyRef": {
            "type": "object",
            "properties": {
              "secret": { "type": "string" },
              "version": { "type": "string" }
            },
            "required": ["secret", "version"]
          }
        }
      }
    },
    "required": ["name"]
  },
  "default": []
}

3. Update the pool with the edited template

prefect work-pool update my-cr-v2 --base-job-template base_v2.json

4. Provide secret-based env via job_variables at deployment time

job_variables = {
  "container_env": [
    { "name": "NAME", "value": "armor-app" },
    {
      "name": "PREFECT_API_URL",
      "valueSource": {
        "secretKeyRef": { "secret": "PREFECT_API_URL", "version": "latest" }
      }
    },
    {
      "name": "PREFECT_API_AUTH_STRING",
      "valueSource": {
        "secretKeyRef": { "secret": "PREFECT_API_AUTH_STRING", "version": "latest" }
      }
    }
  ]
}

Notes

• Ensure the Cloud Run Job’s service account has Secret Manager Secret Accessor (roles/secretmanager.secretAccessor).
• Prefect will still inject required envs (e.g., PREFECT_API_URL, PREFECT__FLOW_RUN_ID). Your container_env is merged alongside.
• If you must remain on the v1 worker, options are limited: create a custom worker that merges a container_env array into containers[].env before submission, or fetch from Secret Manager at runtime in your code and export to os.environ.

References

• Prefect GCP worker guide: https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide
• Customize job variables: https://docs-3.prefect.io/v3/how-to-guides/deployments/customize-job-variables
• Cloud Run: Configure secrets as env vars: https://cloud.google.com/run/docs/configuring/secrets#reference-secret-env
• Cloud Run Job YAML (v1) reference: https://cloud.google.com/run/docs/reference/yaml/v1#googlecloudrunv1job

---

This discussion was automatically created by the Marvin bot to preserve valuable community insights.