HyperFleet E2E Test Runbook

Audience: Developers running e2e tests locally

This runbook provides step-by-step instructions for setting up, running, and troubleshooting HyperFleet E2E tests in a local development environment.

Prerequisites
Prepare Test Environment
Deploy CLM to Your Created GKE Cluster
Running E2E Tests Locally
Common Failure Modes and Troubleshooting
Test Coverage in CI

Prerequisites

Required Tools

The following tools must be installed on your local machine:

Tool	Minimum Version	Purpose	Installation
Go	1.25+	Build and run the E2E framework	go.dev
kubectl	1.28+	Interact with Kubernetes clusters	kubernetes.io
helm	3.0+	Deploy HyperFleet components	helm.sh
git	2.30+	Clone repositories and manage Helm charts	git-scm.com
podman or docker	Latest	Build container images (optional)	podman.io or docker.com

Verify Prerequisites

Run these commands to verify your setup:

# Check Go version
go version  # Should show 1.25 or higher

# Check kubectl
kubectl version --client

# Check Helm
helm version

# Check Git
git --version

# Check container tool (optional)
podman --version || docker --version

Prepare Test Environment

Clone and Configure Terraform

First, clone the infrastructure repository and navigate to the terraform directory:

git clone https://github.com/openshift-hyperfleet/hyperfleet-infra/
cd hyperfleet-infra/terraform

Install GKE Cluster

Run the following Terraform commands to deploy your GKE cluster.

Terraform Commands

# Copy and update the terraform variable file
cp envs/gke/dev.tfvars.example envs/gke/dev-<your name>.tfvars
# Update the following settings in your tfvars file
# developer_name - set to your name, use_pubsub=false, enable_dead_letter=false

# Copy and update the terraform backend file
cp envs/gke/dev.tfbackend.example envs/gke/dev-<your name>.tfbackend
# update the prefix field with your name

# Initialize terraform with your backend configuration
terraform init -backend-config=envs/gke/dev-<your name>.tfbackend

# Preview the infrastructure changes
terraform plan -var-file=envs/gke/dev-<your name>.tfvars

# Apply the infrastructure changes
terraform apply -var-file=envs/gke/dev-<your name>.tfvars

Install Maestro

After deploying the GKE cluster, install Maestro and create a consumer:

# Install Maestro
make install-maestro

# Create Maestro consumer (default: cluster1, test adapter are configured with it)
make create-maestro-consumer MAESTRO_CONSUMER=cluster1

# Patch the service type to LoadBalancer to expose a external IP
kubectl patch svc maestro -n maestro -p '{"spec":{"type":"LoadBalancer"}}'

Login to Cluster

After the deployment completes, log in to the cluster locally using the output command (replace your name):

gcloud container clusters get-credentials hyperfleet-dev-<your name> --zone us-central1-a --project hcm-hyperfleet

Deploy CLM to Your Created GKE Cluster

Clone the Repository

git clone https://github.com/openshift-hyperfleet/hyperfleet-e2e.git
cd hyperfleet-e2e

Deploy HyperFleet Components

The E2E tests require a running HyperFleet environment (API, Sentinel, and Adapters).

# 1. Copy the example configuration
cd deploy-scripts/
cp .env.example .env

# 2. Edit .env with your settings
vim .env
source .env

# 3. Deploy with custom configuration
./deploy-clm.sh --action install --namespace "${NAMESPACE}"

Key Configuration Parameters (in .env):

# GCP configuration (required for Pub/Sub)
export GCP_PROJECT_ID="${GCP_PROJECT_ID:-hcm-hyperfleet}"

# Image configuration (optional - defaults to latest)
export API_IMAGE_TAG="${API_IMAGE_TAG:-latest}"
export SENTINEL_IMAGE_TAG="${SENTINEL_IMAGE_TAG:-latest}"
export ADAPTER_IMAGE_TAG="${ADAPTER_IMAGE_TAG:-latest}"

# Adapters to deploy (optional)
export CLUSTER_TIER0_ADAPTERS_DEPLOYMENT="${CLUSTER_TIER0_ADAPTERS_DEPLOYMENT:-cl-namespace,cl-job,cl-deployment,cl-maestro}"
export NODEPOOL_TIER0_ADAPTERS_DEPLOYMENT="${NODEPOOL_TIER0_ADAPTERS_DEPLOYMENT:-np-configmap}"

# Adapters for API cluster/nodepool configuration
export API_ADAPTERS_CLUSTER="${API_ADAPTERS_CLUSTER:-cl-namespace,cl-job,cl-deployment,cl-maestro}"
export API_ADAPTERS_NODEPOOL="${API_ADAPTERS_NODEPOOL:-np-configmap}"

Verify Deployment

# Check Helm releases
helm list -n "${NAMESPACE}"

# Verify all pods are running
kubectl get pods -n "${NAMESPACE}"

# Check pod logs if any issues
kubectl logs -n "${NAMESPACE}" <pod-name>

Expected State: All pods should show status Running with READY 1/1.

Running E2E Tests Locally

Build the E2E Framework

# Generate API client from OpenAPI spec
make generate

# Build the hyperfleet-e2e binary
make build

# Verify the build
./bin/hyperfleet-e2e --help

Configure API Access

If the Maestro and Hyperfleet API services are not exposed via LoadBalancer, you'll need to port-forward them locally:

# Terminal 1 - Port-forward Maestro API (local port 8000)
kubectl port-forward -n maestro svc/maestro 8000:8000

# Terminal 2 - Port-forward Hyperfleet API (local port 8001)
kubectl port-forward -n ${NAMESPACE} svc/hyperfleet-api 8001:8000

Then configure your environment variables:

export MAESTRO_URL=http://localhost:8000
export HYPERFLEET_API_URL=http://localhost:8001

Basic Test Execution

# Run tests with specific label
./bin/hyperfleet-e2e test --label-filter=tier0

# Run tests for specific suite
./bin/hyperfleet-e2e test --focus "\[Suite: cluster\]"

# Run specific test by description
./bin/hyperfleet-e2e test --focus "Create Cluster via API"

Example:

# Using environment variable
export HYPERFLEET_API_URL=<value>
export MAESTRO_URL=<value>
export NAMESPACE=<NAMESPACE>
# Run all tier0 cases
./bin/hyperfleet-e2e test --label-filter=tier0

# Run all tier1 cases
./bin/hyperfleet-e2e test --label-filter=tier1

View All Options

# Show all available commands
./bin/hyperfleet-e2e --help

# Show test command options
./bin/hyperfleet-e2e test --help

Common Failure Modes and Troubleshooting

Tools and Tips

The following tools are available to help debug and interact with HyperFleet components:

Tool	Purpose	Link
Hyperfleet Explorer	View cluster/nodepool API responses	https://github.com/rh-amarin/hyperfleet-explorer
Scripts	Interact with various component APIs and perform operations	https://github.com/rh-amarin/hyperfleet-scripts
k9s	Kubernetes CLI to manage your clusters in style!	https://k9scli.io/

General Troubleshooting

Namespace Configuration

Important: Set the NAMESPACE environment variable to match the namespace used during deployment. Some test cases deploy adapters dynamically and need to target the same namespace where your HyperFleet components are running.

# Set NAMESPACE if you deployed to a custom namespace
export NAMESPACE=my-custom-namespace
./bin/hyperfleet-e2e test --label-filter=tier0

Timeout Errors

If you encounter timeout errors like this:

[FAILED] cluster creation failed
Unexpected error:
  failed to create cluster: Post "http://34.9.19.133:8000/api/hyperfleet/v1/clusters":
  context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Troubleshooting steps:

Check if all pods are running:

kubectl get pods -n hyperfleet-e2e

Expected output - all pods should show Running with READY 1/1:

NAME                                 READY   STATUS    RESTARTS   AGE
hyperfleet-api-xxx                   1/1     Running   0          10m
hyperfleet-sentinel-xxx              1/1     Running   0          10m
cl-namespace-adapter-xxx             1/1     Running   0          10m
cl-job-adapter-xxx                   1/1     Running   0          10m

Check pod logs for errors:

# Check API logs
kubectl logs -n hyperfleet-e2e deployment/hyperfleet-api --tail=50

# Check Sentinel logs
kubectl logs -n hyperfleet-e2e deployment/hyperfleet-sentinel --tail=50

# Check adapter logs
kubectl logs -n hyperfleet-e2e deployment/cl-namespace-adapter --tail=50

Verify API connectivity:

# Test API endpoint
curl -f -X GET ${HYPERFLEET_API_URL}/api/hyperfleet/v1/clusters/

Check service endpoints:

# Verify LoadBalancer has external IP
kubectl get svc -n hyperfleet-e2e hyperfleet-api

Test Coverage in CI

How Your Tests Run in CI

The test cases you run locally are automatically picked up and executed in nightly CI jobs to ensure continuous validation of the system.

Job Configuration File: All job definitions can be found in the openshift-hyperfleet-hyperfleet-e2e-main__e2e.yaml configuration file.

Job Name	Test Tier	Schedule	Description
tier0-nightly	tier0	Daily	Runs basic smoke tests and happy critical path validations
tier1-nightly	tier1	Daily	Runs extended test suite

Job Configuration and Management

For comprehensive information about CI jobs, see the Add HyperFleet E2E CI Job in Prow documentation, which covers:

How CI jobs are configured in Prow
Viewing job results
Triggering jobs manually
Debugging job failures

Changelog

All notable changes to this document will be documented in this section.

The format is based on Keep a Changelog.

2026-03-30

Added

Initial runbook with prerequisites, environment setup, test execution, troubleshooting, and CI coverage sections
Prerequisites section with required tools and verification steps
Prepare Test Environment section with Terraform and GKE cluster setup
Deploy CLM section with HyperFleet component deployment instructions
Running E2E Tests Locally section with build and execution commands
Common Failure Modes and Troubleshooting section with debugging tools and tips
Test Coverage in CI section documenting nightly jobs and Prow integration

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HyperFleet E2E Test Runbook

Table of Contents

Prerequisites

Required Tools

Verify Prerequisites

Prepare Test Environment

Clone and Configure Terraform

Install GKE Cluster

Terraform Commands

Install Maestro

Login to Cluster

Deploy CLM to Your Created GKE Cluster

Clone the Repository

Deploy HyperFleet Components

Verify Deployment

Running E2E Tests Locally

Build the E2E Framework

Configure API Access

Basic Test Execution

View All Options

Common Failure Modes and Troubleshooting

Tools and Tips

General Troubleshooting

Namespace Configuration

Timeout Errors

Test Coverage in CI

How Your Tests Run in CI

Job Configuration and Management

Changelog

2026-03-30

Added

FilesExpand file tree

runbook.md

Latest commit

History

runbook.md

File metadata and controls

HyperFleet E2E Test Runbook

Table of Contents

Prerequisites

Required Tools

Verify Prerequisites

Prepare Test Environment

Clone and Configure Terraform

Install GKE Cluster

Terraform Commands

Install Maestro

Login to Cluster

Deploy CLM to Your Created GKE Cluster

Clone the Repository

Deploy HyperFleet Components

Verify Deployment

Running E2E Tests Locally

Build the E2E Framework

Configure API Access

Basic Test Execution

View All Options

Common Failure Modes and Troubleshooting

Tools and Tips

General Troubleshooting

Namespace Configuration

Timeout Errors

Test Coverage in CI

How Your Tests Run in CI

Job Configuration and Management

Changelog

2026-03-30

Added