Add a script for the ax harness deployment and update readme.

internal/hack/install-ax.sh

@@ -0,0 +1,112 @@
+#!/bin/bash
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+set -u
+set -o pipefail
+
+ROOT=$(git rev-parse --show-toplevel)
+cd "${ROOT}"
+
+# ANSI color codes for prettier output
+COLOR_CYAN='\033[1;36m'
+COLOR_RESET='\033[0m'
+
+function log_step() {
+  local step_name="$1"
+  echo -e "${COLOR_CYAN}[step]: ${step_name}${COLOR_RESET}"
+}
+
+function usage() {
+  echo "Usage: $0 [options]"
+  echo ""
+  echo "Options:"
+  echo "  --deploy-ax-server                    Deploy AX server and components using ko"
+  echo "  --delete-ax-server                    Delete AX server and components from cluster"
+  echo "  -h, --help                            Show this help message"
+}
+
+run_kubectl() {
+  kubectl \
+    ${KUBECTL_CONTEXT:+--context=${KUBECTL_CONTEXT}} \
+    "$@"
+}
+
+run_ko() {
+  GOFLAGS="-tags=harness" ko apply \
+    ${KUBECTL_CONTEXT:+--context=${KUBECTL_CONTEXT}} \
+    "$@"
+}
+
+deploy_ax_server() {
+  log_step "deploy_ax_server"
+
+  # Check dependencies
+  if [[ -z "${GEMINI_API_KEY:-}" ]]; then
+    echo "Error: GEMINI_API_KEY environment variable must be set" >&2
+    exit 1
+  fi
+  if [[ -z "${BUCKET_NAME:-}" ]]; then
+    echo "Error: BUCKET_NAME environment variable must be set" >&2
+    exit 1
+  fi
+
+  echo "Using GCS Bucket: ${BUCKET_NAME}"
+
+  # Render template and apply with ko
+  sed -e "s|\${GEMINI_API_KEY}|${GEMINI_API_KEY}|g" \
+      -e "s|\${BUCKET_NAME}|${BUCKET_NAME}|g" \
+      internal/manifests/ax-deployment2.yaml \
+      | run_ko -f -
+}
+
+delete_ax_server() {
+  log_step "delete_ax_server"
+
+  # Delete resources using a dummy key and bucket so credentials aren't required for deletion
+  sed -e "s|\${GEMINI_API_KEY}|dummy-key|g" \
+      -e "s|\${BUCKET_NAME}|dummy-bucket|g" \
+      internal/manifests/ax-deployment2.yaml \
+      | run_kubectl delete --ignore-not-found -f -
+}
+
+if [ "$#" -eq 0 ]; then
+  usage
+  exit 1
+fi
+
+# If -h or --help appears anywhere in the command line, print the usage and exit.
+for arg in "$@"; do
+  case "$arg" in
+    -h|--help)
+      usage
+      exit 0
+      ;;
+  esac
+done
+
+while [[ "$#" -gt 0 ]]; do
+  case $1 in
+    --deploy-ax-server) deploy_ax_server ;;
+    --delete-ax-server) delete_ax_server ;;
+    *)
+      echo "Error: unknown option: $1" >&2
+      echo ""
+      usage
+      exit 1
+      ;;
+  esac
+  shift
+done

internal/manifests/README.md

@@ -1,71 +1,96 @@
 # AX Harness Deployment on Kubernetes
 
-This directory contains manifests to deploy AX with the `harness` configuration path on Kubernetes using Agent Substrate.
+> [!WARNING]
+> 🚧 **The `harness` deployment path is under active development.**
+>
+> This path is experimental and incomplete: the manifests, scripts, and
+> runtime behavior will change and may break without notice.
+
+This directory contains Kubernetes manifests and configurations to deploy
+and verify the AX `harness` configuration path on Kubernetes using Agent
+Substrate.
+
+The target Kubernetes cluster is assumed to have
+[Agent Substrate](https://github.com/agent-substrate/substrate) installed.
 
 ---
 
-## Deploying
+## 🚀 Deploying to Agent Substrate
+
+This deploys the AX `harness` path: the AX harness `WorkerPool` and `ActorTemplate` — provisioned as isolated, warm-standby actors that are live-snapshotted on boot and instantly restored from GCS when a new conversation starts — together with an `ax-server` controller front-end (a `ReplicaSet`).
+
+### 1. Build and Deploy
 
-Make sure your target environment variables are exported:
+> [!NOTE]
+> Do not manually edit `internal/manifests/ax-deployment2.yaml`. The installation script automatically injects your `${GEMINI_API_KEY}` and `${BUCKET_NAME}` environment variables during deployment.
+
+Use the installation script to build the images (with the `harness` build tag) and apply the resolved manifests to your cluster:
 
 ```bash
 export PROJECT_ID="ax-substrate" # Your GCP project ID
 export GEMINI_API_KEY="your-api-key"
 export BUCKET_NAME="snapshot-substrate-test-$PROJECT_ID"
 export KO_DOCKER_REPO="gcr.io/$PROJECT_ID/ate-images"
 export KO_DEFAULTPLATFORMS="linux/amd64"
+
+./internal/hack/install-ax.sh --deploy-ax-server
 ```
 
-Render the template variables and apply the resolved manifest:
+This command will:
+- Build the AX images using `ko` with the `harness` build tag.
+- Create the `ax` namespace.
+- Create the `WorkerPool` and `ActorTemplate` for the AX harness.
+- Create the `ax-server` `ReplicaSet` (the controller front-end).
 
+Wait until the template is ready:
 ```bash
-sed -e "s|\${GEMINI_API_KEY}|${GEMINI_API_KEY}|g" \
-    -e "s|\${BUCKET_NAME}|${BUCKET_NAME}|g" \
-    internal/manifests/ax-deployment2.yaml \
-    | GOFLAGS="-tags=harness" ko apply -f -
+kubectl wait --for=condition=Ready actortemplate/ax-harness-template -n ax --timeout=5m
 ```
 
-Use the following command for initial deployment:
+### 2. Port-Forward Services
+
+The `harness` path has no Envoy router or `Service`; connect directly to the `ax-server` `ReplicaSet`:
 
 ```bash
-kubectl apply -f ax-deployment2.yaml
+# Port-forward the ax-server ReplicaSet
+kubectl port-forward -n ax rs/ax-server 8494:8494
 ```
 
-If you have a deployment and rolling out a new version, you can do:
+### 3. Test End-to-End
+
+Run an execution targeting the port-forwarded server:
 
 ```bash
-kubectl delete pods -l app=ax-server -n ax 
-kubectl delete workerpool ax-harness-workerpool -n ax
-kubectl delete actortemplate ax-harness-template -n ax
+ax exec --server=localhost:8494 --input="hello"
 ```
 
-Wait until all pods are up and running:
+The server should respond with:
+```text
+Conversation: fb344a18-3720-4c4f-8a6e-2ce34db975b3
 
-```bash
-kubectl get pods -n ax
-```
+⏺ hello
 
----
+Hello world
+```
+*The request is served directly by the `ax-server`.*
 
-## Testing the Deployment
+## 🧹 How to Uninstall
 
-Proxy the `ax-server` ReplicaSet port to your local environment:
+To remove AX resources from your cluster, run:
 
 ```bash
-kubectl port-forward -n ax rs/ax-server 8494:8494
+./internal/hack/install-ax.sh --delete-ax-server
 ```
 
-Execute a query against the local port-forwarded server using the `ax` CLI:
+---
 
-```bash
-ax exec --server localhost:8494 --input "hello"
-```
+## 🛠️ Inspection & Diagnostics
 
-The server should respond with:
-```text
-Conversation: fb344a18-3720-4c4f-8a6e-2ce34db975b3
+Use the **`kubectl ate`** CLI tool to inspect the live states of
+active actors and allocated standby worker pool instances:
 
-⏺ hello
+```bash
+kubectl ate get actors
 
-Hello world
+kubectl ate get workers
 ```