[Deployment] Dynamic Versioning, Release Workflow Split, and CI/CD Hardening

Dockerfile

@@ -0,0 +1,52 @@
+# Copyright 2025 Google LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM python:3.12-slim
+
+ENV PYTHONUNBUFFERED=1
+
+# Install curl for healthcheck
+RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+# Build arguments for version and index (default to standard PyPI)
+ARG MCP_VERSION
+ARG PIP_INDEX_URL=https://pypi.org/simple/
+ARG PIP_EXTRA_INDEX_URL=https://pypi.org/simple/
+
+# Check if MCP_VERSION is set
+RUN if [ -z "$MCP_VERSION" ]; then echo "MCP_VERSION is not set" && exit 1; fi
+
+# Install packages in a single layer
+# 1. Pre-install fastapi from PyPI to avoid TestPyPI squatting
+# 2. Install main package
+# Note: We must explicitly unset PIP_EXTRA_INDEX_URL for the first command to force PyPI usage.
+RUN PIP_EXTRA_INDEX_URL="" pip install --no-cache-dir "fastapi>=0.115.0" --index-url https://pypi.org/simple/ && \
+    pip install --no-cache-dir \
+    --index-url ${PIP_INDEX_URL} \
+    --extra-index-url ${PIP_EXTRA_INDEX_URL} \
+    datacommons-mcp==${MCP_VERSION}
+
+# Create non-root user with explicit UID/GID and disabled shell
+RUN groupadd --gid 1001 mcp && useradd -m --uid 1001 --gid 1001 --shell /bin/false mcp
+USER mcp
+
+ENV PORT=8080
+
+# Health check with Accept header and explicit PORT
+HEALTHCHECK CMD curl --fail -H "Accept: application/json" http://localhost:${PORT}/health || exit 1
+
+# Use sh -c for variable expansion
+CMD ["sh", "-c", "datacommons-mcp serve http --host 0.0.0.0 --port ${PORT}"]

deploy/README.md

@@ -0,0 +1,88 @@
+# Deployment Guide
+
+This repository uses **Google Cloud Build** for CI/CD, with three distinct deployment tiers.
+
+## 1. Autopush (Development)
+- **Trigger**: Push to `main` branch.
+- **Config**: `deploy/autopush.yaml`
+- **Output**:
+  - **PyPI**: `datacommons-mcp` (TestPyPI) version `X.Y.Z.devN` (Sequential dev versions).
+  - **Docker**: 
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:autopush-X.Y.Z.devN` - immutable
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:autopush` - latest autopush
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:latest` - latest overall
+  - **Cloud Run**: `mcp-server-autopush` (Auto-updated).
+- **Purpose**: Rapid testing of the latest code on the `main` branch.
+
+## 2. Staging (Release Candidates)
+- **Trigger**: Pushing a tag matching `v*` (specifically `rc` tags like `v1.1.3rc1`).
+- **Config**: `deploy/staging.yaml`
+- **Output**:
+  - **PyPI**: `datacommons-mcp` (TestPyPI) version `X.Y.ZrcN`.
+  - **Docker**: 
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:staging-vX.Y.ZrcN` - immutable
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:staging` - latest staging
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:latest` - latest overall
+  - **Cloud Run**: `mcp-server-staging` (Pinned to such tag).
+- **Purpose**: Verifying releases in a production-like environment before going live.
+
+### How to Create a Staging Release
+Run the helper script to automatically find the next available RC version and push the tag:
+```bash
+python3 scripts/create_staging_tag.py
+```
+Or manually:
+```bash
+git tag v1.1.3rc1
+git push origin v1.1.3rc1
+```
+
+## 3. Production Release
+- **Trigger**: Pushing a tag matching `v*` that is **NOT** an `rc` (e.g., `v1.1.3`).
+- **Config**: `deploy/release.yaml`
+- **Output**:
+  - **PyPI**: `datacommons-mcp` (**Official PyPI**) version `X.Y.Z`.
+  - **Docker**: 
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:production-vX.Y.Z` - immutable
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:production` - latest production
+    - `gcr.io/$PROJECT_ID/datacommons-mcp-server:latest` - latest overall
+  - **Cloud Run**: `mcp-server-prod` (Pinned to such tag).
+- **Purpose**: Official public release to PyPI and Production Cloud Run.
+
+> [!NOTE]
+> The `:latest` tag is pushed by **production** pipeline only. It always points to the single most recently built image deployed to prod.
+
+### Production Release Process
+The process to release to production is a 2-step workflow: **Prepare** (Version Bump) -> **Release** (Tag & Deploy).
+
+#### Step 1: Version Bump (Prepare)
+Run this script to calculate the next version, update `pyproject.toml`, and create a PR.
+
+```bash
+python3 scripts/create_release_pr.py
+# Follow the interactive prompt (Major/Minor/Patch)
+```
+
+1.  This triggers a Cloud Build job (`deploy/bump_version.yaml`).
+2.  A Pull Request will be created (e.g., `chore: bump version to 1.1.4`).
+3.  **Review and Merge** this PR into `main`.
+
+#### Step 2: Deploy (Release)
+Once the version bump is merged, create the official release to trigger deployment.
+
+**Using GitHub UI (Recommended)**
+1.  Go to [Draft a New Release](https://github.com/datacommonsorg/agent-toolkit/releases/new).
+2.  **Choose a tag**: Create a new tag matching your bumped version (e.g., `v1.1.4`).
+    *   *Critical: Must match the version you just merged into `pyproject.toml`.*
+3.  **Target**: `main`.
+4.  **Release title**: `v1.1.4`.
+5.  **Description**: Generate release notes.
+6.  Click **Publish release**.
+
+**Or Manual Git Tag**
+```bash
+git checkout main
+git pull
+git tag v1.1.4
+git push origin v1.1.4
+```

deploy/autopush.yaml

@@ -0,0 +1,82 @@
+# Copyright 2025 Google LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+steps:
+  # 1. Install uv and build tools
+  - name: python:3.12-slim
+    entrypoint: bash
+    args:
+      - -c
+      - |
+        pip install uv
+        # Generate dynamic version and update version.py
+        # Use python script to get next sequential dev version from TestPyPI
+        new_version=$(python3 scripts/get_next_version.py --type dev)
+        echo "$new_version" > _version.txt
+        echo "Generated version: $new_version"
+        sed -i 's/^version = ".*"/version = "'$new_version'"/' packages/datacommons-mcp/pyproject.toml
+
+        # Build and Publish
+        cd packages/datacommons-mcp
+        uv build --out-dir dist
+        uv publish --publish-url https://test.pypi.org/legacy/ --token $$TEST_PYPI_TOKEN
+    secretEnv: ['TEST_PYPI_TOKEN']
+
+  # 2. Wait for TestPyPI propagation
+  - name: python:3.12-slim
+    entrypoint: bash
+    args:
+    - -c
+    - |
+      pip install certifi
+      python3 scripts/wait_for_pypi.py datacommons-mcp $(cat _version.txt) --repository https://test.pypi.org/simple
+
+  # 3. Build Container Image
+  - name: gcr.io/cloud-builders/docker
+    entrypoint: bash
+    args:
+      - -c
+      - |
+        docker build \
+          --build-arg MCP_VERSION=$(cat _version.txt) \
+          --build-arg PIP_INDEX_URL=https://pypi.org/simple/ \
+          --build-arg PIP_EXTRA_INDEX_URL=https://test.pypi.org/simple/ \
+          -t gcr.io/$PROJECT_ID/datacommons-mcp-server:autopush \
+          -t gcr.io/$PROJECT_ID/datacommons-mcp-server:autopush-$(cat _version.txt) .
+        docker push gcr.io/$PROJECT_ID/datacommons-mcp-server:autopush
+        docker push gcr.io/$PROJECT_ID/datacommons-mcp-server:autopush-$(cat _version.txt)
+
+  # 4. Deploy to Cloud Run
+  - name: gcr.io/cloud-builders/gcloud
+    entrypoint: bash
+    args:
+      - -c
+      - |
+        gcloud run deploy mcp-server-autopush \
+        --image=gcr.io/$PROJECT_ID/datacommons-mcp-server:autopush \
+        --region=us-central1 \
+        --platform=managed \
+        --no-allow-unauthenticated \
+        --set-env-vars=MCP_VERSION=$(cat _version.txt) \
+        --set-secrets=DC_API_KEY=dc-api-key-for-mcp:latest \
+        --service-account=datacommons-mcp-server@datcom-mixer-autopush.iam.gserviceaccount.com \
+        --project=datcom-mixer-autopush
+
+availableSecrets:
+  secretManager:
+  - versionName: projects/$PROJECT_ID/secrets/TEST_PYPI_TOKEN/versions/latest
+    env: 'TEST_PYPI_TOKEN'
+
+options:
+  logging: CLOUD_LOGGING_ONLY

deploy/bump_version.yaml

@@ -0,0 +1,63 @@
+steps:
+  - name: python:3.12-slim
+    entrypoint: bash
+    secretEnv: ['GH_TOKEN']
+    env: ['NEW_VERSION=${_NEW_VERSION}']
+    args:
+      - -c
+      - |
+        apt-get update && apt-get install -y git curl gh
+
+        # Fail on error
+        set -e
+
+        # Config Git with Robot Author
+        git config --global url."https://${_GITHUB_AUTHOR}:$$GH_TOKEN@github.com/".insteadOf "https://github.com/"
+        git config --global credential.helper store
+        echo "https://${_GITHUB_AUTHOR}:$$GH_TOKEN@github.com" > /root/.git-credentials
+        chmod 600 /root/.git-credentials
+
+        git config --global user.email "${_GITHUB_AUTHOR}@users.noreply.github.com"
+        git config --global user.name "${_GITHUB_AUTHOR}"
+
+        # Clone Repo
+        rm -rf /workspace/* /workspace/.[!.]*
+        git clone "https://github.com/${_GITHUB_ORG}/${_GITHUB_REPO}.git" /workspace --branch "${_GITHUB_BRANCH}" --single-branch --depth 1
+
+        # Create Branch
+        BRANCH_NAME="chore/bump-version-$$NEW_VERSION"
+        git checkout -b $$BRANCH_NAME
+
+        # Update Version File
+        echo "Updating pyproject.toml to $$NEW_VERSION"
+        grep "version =" packages/datacommons-mcp/pyproject.toml || echo "Version line not found!"
+        sed -i 's/^version = ".*"/version = "'$$NEW_VERSION'"/' packages/datacommons-mcp/pyproject.toml
+        grep "version =" packages/datacommons-mcp/pyproject.toml
+
+        # Commit & Push
+        git add packages/datacommons-mcp/pyproject.toml
+        git commit -m "chore: bump version to $$NEW_VERSION"
+        git push origin $$BRANCH_NAME
+
+        # Create PR
+        echo $$GH_TOKEN | gh auth login --with-token
+        gh pr create \
+          --repo "${_GITHUB_ORG}/${_GITHUB_REPO}" \
+          --title "chore: bump version to $$NEW_VERSION" \
+          --body "Automated PR from Cloud Build release pipeline matching to update our version to $$NEW_VERSION ahead of its release" \
+          --base "${_GITHUB_BRANCH}" \
+          --head "$$BRANCH_NAME" || echo "PR creation failed, it might already exist."
+
+substitutions:
+  _GITHUB_AUTHOR: 'datacommons-robot-author'
+  _GITHUB_ORG: 'datacommonsorg'
+  _GITHUB_REPO: 'agent-toolkit'
+  _GITHUB_BRANCH: 'main'
+
+availableSecrets:
+  secretManager:
+  - versionName: projects/$PROJECT_ID/secrets/agent-toolkit-github-pat/versions/latest
+    env: 'GH_TOKEN'
+
+options:
+  logging: CLOUD_LOGGING_ONLY

deploy/release.yaml

@@ -0,0 +1,85 @@
+# Copyright 2025 Google LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+steps:
+  # 1. Publish to PyPI
+  - name: python:3.12-slim
+    entrypoint: bash
+    env: ['TAG_NAME=$TAG_NAME']
+    args:
+      - -c
+      - |
+        pip install uv
+        tag_version=$${TAG_NAME#v}
+        echo "$tag_version" > _version.txt
+
+        # Inject Tag Version into pyproject.toml (Safety override)
+        sed -i 's/^version = ".*"/version = "'$tag_version'"/' packages/datacommons-mcp/pyproject.toml
+
+        cd packages/datacommons-mcp
+        uv build --out-dir dist
+        # Prod PyPI
+        uv publish --token $$PYPI_TOKEN
+    secretEnv: ['PYPI_TOKEN']
+
+  # 2. Wait
+  - name: python:3.12-slim
+    entrypoint: bash
+    args:
+    - -c
+    - |
+      pip install certifi
+      python3 scripts/wait_for_pypi.py datacommons-mcp $(cat _version.txt)
+
+  # 3. Build & Push
+  - name: gcr.io/cloud-builders/docker
+    entrypoint: bash
+    args:
+      - -c
+      - |
+        # Standard PyPI (default args)
+        docker build \
+          --build-arg MCP_VERSION=$(cat _version.txt) \
+          -t gcr.io/$PROJECT_ID/datacommons-mcp-server:production \
+          -t gcr.io/$PROJECT_ID/datacommons-mcp-server:$TAG_NAME \
+          -t gcr.io/$PROJECT_ID/datacommons-mcp-server:production-$TAG_NAME \
+          -t gcr.io/$PROJECT_ID/datacommons-mcp-server:latest .
+        docker push gcr.io/$PROJECT_ID/datacommons-mcp-server:production
+        docker push gcr.io/$PROJECT_ID/datacommons-mcp-server:$TAG_NAME
+        docker push gcr.io/$PROJECT_ID/datacommons-mcp-server:production-$TAG_NAME
+        docker push gcr.io/$PROJECT_ID/datacommons-mcp-server:latest
+
+  # 4. Deploy
+  - name: gcr.io/cloud-builders/gcloud
+    entrypoint: bash
+    args:
+      - -c
+      - |
+        gcloud run deploy mcp-server-prod \
+        --image=gcr.io/$PROJECT_ID/datacommons-mcp-server:production \
+        --region=us-central1 \
+        --platform=managed \
+        --no-allow-unauthenticated \
+        --set-env-vars=MCP_VERSION=$(cat _version.txt) \
+        --set-secrets=DC_API_KEY=dc-api-key-for-mcp:latest \
+        --service-account=datacommons-mcp-server@datcom-mixer.iam.gserviceaccount.com \
+        --project=datcom-mixer
+
+availableSecrets:
+  secretManager:
+  - versionName: projects/$PROJECT_ID/secrets/PYPI_TOKEN/versions/latest
+    env: 'PYPI_TOKEN'
+
+options:
+  logging: CLOUD_LOGGING_ONLY