Skip to content

fix: improve pipeline reliability with image verification and retry logic #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fok666 merged 4 commits into from
Jan 18, 2026
Merged

fix: improve pipeline reliability with image verification and retry logic #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
870a1f6
docs: add available profiles matrix to README
Jan 18, 2026
80db199
fix: resolve code scan security vulnerabilities
Jan 18, 2026
3896c83
fix: address PR review feedback
Jan 18, 2026
073acc2
fix: improve pipeline reliability with image verification and retry l…
fok666 Jan 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 46 additions & 6 deletions .github/workflows/docker-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,27 @@ jobs:
cache-to: type=gha,mode=max,scope=${{ matrix.profile.name }}-${{ matrix.platform }}
sbom: true
provenance: true
continue-on-error: true

- name: Verify image was pushed
run: |
echo "Verifying image exists in registry..."
IMAGE="${{ env.GHCR_IMAGE }}:${{ steps.versions.outputs.agent }}-ubuntu${{ steps.versions.outputs.os }}-${{ matrix.profile.name }}-${{ steps.date.outputs.date }}-${{ matrix.platform }}"

# Wait a bit for the image to be available in the registry
sleep 10

# Retry up to 3 times
for i in {1..3}; do
if docker buildx imagetools inspect "$IMAGE" > /dev/null 2>&1; then
echo "✓ Image verified: $IMAGE"
exit 0
fi
echo "Attempt $i: Image not yet available, waiting..."
sleep 10
done

echo "✗ Failed to verify image: $IMAGE"
exit 1

create-manifest:
name: Create Multi-Arch Manifest (${{ matrix.profile.name }})
Expand Down Expand Up @@ -428,11 +448,31 @@ jobs:
# Clean up disk space before creating manifests
docker system prune -af

# All profiles: amd64 + arm64
docker buildx imagetools create \
-t ${{ env.GHCR_IMAGE }}:${{ steps.versions.outputs.agent }}-ubuntu${{ steps.versions.outputs.os }}-${{ matrix.profile.name }}-${{ steps.date.outputs.date }} \
${{ env.GHCR_IMAGE }}:${{ steps.versions.outputs.agent }}-ubuntu${{ steps.versions.outputs.os }}-${{ matrix.profile.name }}-${{ steps.date.outputs.date }}-amd64 \
${{ env.GHCR_IMAGE }}:${{ steps.versions.outputs.agent }}-ubuntu${{ steps.versions.outputs.os }}-${{ matrix.profile.name }}-${{ steps.date.outputs.date }}-arm64
# Verify both images exist before creating manifest
AMD64_IMAGE="${{ env.GHCR_IMAGE }}:${{ steps.versions.outputs.agent }}-ubuntu${{ steps.versions.outputs.os }}-${{ matrix.profile.name }}-${{ steps.date.outputs.date }}-amd64"
ARM64_IMAGE="${{ env.GHCR_IMAGE }}:${{ steps.versions.outputs.agent }}-ubuntu${{ steps.versions.outputs.os }}-${{ matrix.profile.name }}-${{ steps.date.outputs.date }}-arm64"

echo "Verifying source images exist..."
docker buildx imagetools inspect "$AMD64_IMAGE"
docker buildx imagetools inspect "$ARM64_IMAGE"

# Create multi-arch manifest with retry logic
MANIFEST_TAG="${{ env.GHCR_IMAGE }}:${{ steps.versions.outputs.agent }}-ubuntu${{ steps.versions.outputs.os }}-${{ matrix.profile.name }}-${{ steps.date.outputs.date }}"

for i in {1..3}; do
if docker buildx imagetools create \
-t "$MANIFEST_TAG" \
"$AMD64_IMAGE" \
"$ARM64_IMAGE"; then
echo "✓ Manifest created successfully: $MANIFEST_TAG"
exit 0
fi
echo "Attempt $i failed, retrying in 15 seconds..."
sleep 15
done

echo "✗ Failed to create manifest after 3 attempts"
exit 1

- name: Create additional tags
run: |
Expand Down
19 changes: 13 additions & 6 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,9 @@ RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes \
&& rm -rf /var/lib/apt/lists/*

# Install sudo...
# SECURITY NOTE: NOPASSWD:ALL is configured for CI/CD automation purposes.
# This allows the agent user to execute commands with sudo without password prompts.
# This is a security trade-off for CI/CD runner functionality.
RUN test "${ADD_SUDO}" = "1" || exit 0 && \
apt-get update && apt-get install -y --no-install-recommends sudo \
&& apt clean \
Expand Down Expand Up @@ -106,7 +109,9 @@ RUN test "${ADD_JQ}" = "1" || exit 0 && \

# Install latest Azure CLI https://learn.microsoft.com/cli/azure/install-azure-cli-linux
RUN test "${ADD_AZURE_CLI}" = "1" || exit 0 && \
curl -sLS "https://aka.ms/InstallAzureCLIDeb" | bash \
curl -sLS "https://aka.ms/InstallAzureCLIDeb" -o /tmp/install-azure-cli.sh \
&& bash /tmp/install-azure-cli.sh \
&& rm /tmp/install-azure-cli.sh \
Comment on lines +112 to +114
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

This is a great security improvement. Piping curl directly to bash can be risky as it executes remote code without inspection. Downloading the script to a temporary file before executing it is a much safer practice. It also improves debuggability if the download or script execution fails.

&& apt clean \
&& rm -rf /var/lib/apt/lists/* \
&& az config set extension.use_dynamic_install=yes_without_prompt \
Expand Down Expand Up @@ -175,11 +180,12 @@ RUN test "${ADD_OPENTOFU}" = "1" || exit 0 && \

# Instal Terraspace https://terraspace.cloud/docs/install/
RUN test "${ADD_TERRASPACE}" = "1" || exit 0 && \
curl -sL https://apt.boltops.com/boltops-key.public | apt-key add - \
&& echo "deb https://apt.boltops.com stable main" > /etc/apt/sources.list.d/boltops.list \
curl -sL https://apt.boltops.com/boltops-key.public | gpg --dearmor -o /usr/share/keyrings/boltops-archive-keyring.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/boltops-archive-keyring.gpg] https://apt.boltops.com stable main" > /etc/apt/sources.list.d/boltops.list \
Comment on lines +183 to +184
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

Excellent change! apt-key is deprecated and its usage is discouraged for security reasons. Switching to gpg --dearmor and storing the key in /usr/share/keyrings is the modern, recommended, and more secure way to handle repository keys. This prevents the key from being trusted for all repositories on the system.

&& apt-get update \
&& apt-get install -y terraspace \
&& apt clean
&& apt clean \
&& rm -rf /var/lib/apt/lists/*

# Install HELM https://helm.sh/docs/intro/install/
RUN test "${ADD_HELM}" = "1" || exit 0 && \
Expand All @@ -192,9 +198,10 @@ RUN test "${ADD_HELM}" = "1" || exit 0 && \

# Install Kustomize https://kubectl.docs.kubernetes.io/installation/kustomize/
RUN test "${ADD_KUSTOMIZE}" = "1" || exit 0 && \
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash \
curl -sLf "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" -o /tmp/install_kustomize.sh \
&& bash /tmp/install_kustomize.sh \
&& install -o root -g root -m 0755 kustomize /usr/local/bin/kustomize \
&& rm -f kustomize
&& rm -f kustomize /tmp/install_kustomize.sh
Comment on lines +201 to +204
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

This change significantly improves security by avoiding the curl | bash anti-pattern. Downloading the script to a temporary file before execution is the recommended and safer approach. Great job on applying this consistently.


# Install GitHub Runner
WORKDIR /runner
Expand Down
14 changes: 14 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,20 @@ Supported `--build-arg` variables are listed below to easily configure the runne
- `ADD_TERRASPACE`: Installs `terraspace` tool
- `ADD_SUDO`: Installs and enables `sudo` for the runner user group


## Available Profiles

Pre-configured profiles are available for different use cases:

| Profile | Description | Included Tools |
|---------|-------------|----------------|
| **full** | Complete toolset with all available tools | Docker, Azure CLI, AWS CLI, PowerShell (with Azure & AWS modules), kubectl, kubelogin, kustomize, Helm, jq, yq, Terraform, OpenTofu, Terraspace |
| **minimal** | Lightweight profile with essential tools only | Docker, jq, yq |
| **k8s** | Kubernetes-focused profile | Docker, kubectl, kubelogin, kustomize, Helm, jq, yq |
| **iac** | Infrastructure as Code profile with bash-based tools | Docker, Azure CLI, AWS CLI, Terraform, OpenTofu, Terraspace, jq, yq |
| **iac-pwsh** | Infrastructure as Code profile with PowerShell support | Docker, Azure CLI, AWS CLI, PowerShell (with Azure & AWS modules), Terraform, OpenTofu, Terraspace, jq, yq |


# References


Expand Down
2 changes: 1 addition & 1 deletion ec2_monitor.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ echo "Checking for EC2 spot instance termination notice..."

# Query EC2 Instance Metadata Service for spot termination notice
# Returns HTTP 404 if no termination notice is present
RESPONSE=$(curl -s -f "$METADATA_ENDPOINT" 2>/dev/null || echo "")
RESPONSE=$(curl -sf "$METADATA_ENDPOINT" 2>/dev/null || echo "")

# Check if termination notice exists
if [ -n "$RESPONSE" ]; then
Expand Down
6 changes: 5 additions & 1 deletion run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ echo "CPUs per runner: $MAX_CPU"
echo ""

# Launch runners
for R in $(seq 1 $RUNNER_COUNT); do
for R in $(seq 1 "$RUNNER_COUNT"); do
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Good job quoting the $RUNNER_COUNT variable. This prevents potential word splitting and globbing issues, making the script more robust. This change aligns with the repository's style guide.

References
  1. Variables should be quoted to prevent word splitting and globbing. This is from line 8 of the repository style guide. (link)

RUNNER_NAME="runner-$(hostname)-$R"
WORK_DIR="/mnt/runner${R}/_work"
CONTAINER_NAME="github-runner-$R"
Expand All @@ -92,6 +92,10 @@ for R in $(seq 1 $RUNNER_COUNT); do
fi

# Run GitHub runner container
# SECURITY NOTE: --privileged mode grants extended privileges to the container.
# This is required for Docker-in-Docker but poses security risks.
# Consider using rootless Docker or Docker socket mounting as alternatives.
# If --privileged is not needed for your use case, remove this flag.
docker run \
--privileged \
--tty \
Expand Down
6 changes: 3 additions & 3 deletions start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ if [ -z "$GITHUB_TOKEN_FILE" ]; then
fi

GITHUB_TOKEN_FILE=/runner/.token
echo -n $GITHUB_TOKEN > "$GITHUB_TOKEN_FILE"
echo -n "$GITHUB_TOKEN" > "$GITHUB_TOKEN_FILE"
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Excellent! Quoting the $GITHUB_TOKEN variable is a crucial security and robustness improvement. It prevents potential issues like word splitting or globbing if the token were to contain special characters. This adheres to the shell scripting best practices outlined in the repository style guide.

References
  1. Variables, especially those containing secrets, must be quoted to prevent unexpected behavior and potential security risks. This is from line 8 of the repository style guide. (link)

fi

unset GITHUB_TOKEN
Expand All @@ -31,7 +31,7 @@ cleanup() {
# If the agent has some running jobs, the configuration removal process will fail.
# So, give it some time to finish the job.
while true; do
./config.sh remove --token $(cat "$GITHUB_TOKEN_FILE") && break
./config.sh remove --token "$(cat "$GITHUB_TOKEN_FILE")" && break
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Good catch! Quoting the command substitution $(cat "$GITHUB_TOKEN_FILE") ensures that the token is passed as a single argument to the remove command, even if it contains whitespace or other special characters. This is a great improvement for script robustness and aligns with the style guide.

References
  1. Command substitutions should be quoted to ensure their output is treated as a single string, which is critical when dealing with secrets or file contents. This is from line 8 of the repository style guide. (link)


echo "Retrying in 30 seconds..."
sleep 30
Expand All @@ -53,7 +53,7 @@ print_header "1. Configuring GitHub Runner..."
./config.sh --unattended \
--name "${RUNNER_NAME:-$(hostname)}" \
--url "$GITHUB_URL" \
--token $(cat "$GITHUB_TOKEN_FILE") \
--token "$(cat "$GITHUB_TOKEN_FILE")" \
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Another great fix. Quoting the command substitution for the token is essential for script reliability, preventing issues if the token contains special characters. This change correctly follows the repository's style guide on quoting variables and command substitutions.

References
  1. Command substitutions should be quoted to ensure their output is treated as a single string, which is critical when dealing with secrets or file contents. This is from line 8 of the repository style guide. (link)

--labels "${RUNNER_LABELS:-default}" \
--work "${RUNNER_WORK_DIRECTORY:-_work}" \
--replace
Expand Down
9 changes: 6 additions & 3 deletions vmss_monitor.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ STOP_SCRIPT='/opt/stop.sh'
echo "Checking for VMSS scheduled events..."

# Query Azure Instance Metadata Service for scheduled events
curl -s "$METADATA_ENDPOINT" -H 'Metadata: true' > "$EVENTS_FILE"
curl -sf "$METADATA_ENDPOINT" -H 'Metadata: true' > "$EVENTS_FILE"

# Check if termination event is scheduled
if grep -q "Terminate" "$EVENTS_FILE"; then
Expand All @@ -30,9 +30,12 @@ if grep -q "Terminate" "$EVENTS_FILE"; then
EventId=$(jq -r '.Events[] | select(.EventType == "Terminate") | .EventId' "$EVENTS_FILE")
if [ -n "$EventId" ]; then
echo "Acknowledging event: $EventId"
curl -s -X POST "$METADATA_ENDPOINT" \
# Use jq to safely construct JSON payload to handle special characters
JSON_PAYLOAD=$(jq -n --arg eventId "$EventId" '{"StartRequests": [{"EventId": $eventId}]}')
curl -sf -X POST "$METADATA_ENDPOINT" \
-H 'Metadata: true' \
-d "{\"StartRequests\": [{\"EventId\": \"${EventId}\"}]}"
-H 'Content-Type: application/json' \
-d "$JSON_PAYLOAD"
Comment on lines +33 to +38
Copy link

@gemini-code-assist gemini-code-assist bot Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

This is an excellent improvement for script robustness and security. Using jq to construct the JSON payload is much safer than manually creating the JSON string. It correctly handles any special characters that might be present in the EventId, preventing potential errors or injection vulnerabilities.

echo "Event acknowledged successfully"
fi
else
Expand Down