Skip to content

HIVE-29463: Publish nightly Docker images #6387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
okumin wants to merge 2 commits into
base: master
Choose a base branch
from
+23 −10
Open

HIVE-29463: Publish nightly Docker images #6387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
01ce54c
Move the GitHub Action
okumin Mar 13, 2026
99dd998
HIVE-29463: Publish nightly Docker images
okumin Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions .github/workflows/docker-GA-images.yml → .github/workflows/docker-images.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ name: Build and Publish docker images for Hive GA

on:
create:
schedule:
- cron: '17 3 * * *'
Copy link
Copy Markdown
Contributor Author

@okumin okumin Mar 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose an odd time, following the guidance.

The schedule event can be delayed during periods of high loads of GitHub Actions workflow runs. High load times include the start of every hour. If the load is sufficiently high enough, some queued jobs may be dropped. To decrease the chance of delay, schedule your workflow to run at a different time of the hour.

workflow_dispatch:
inputs:
hiveVersion:
Expand All @@ -40,7 +42,7 @@ env:

jobs:
buildAndPush:
if: github.event_name == 'workflow_dispatch' || github.event_name == 'create' && github.event.ref_type == 'tag' && startsWith(github.event.ref, 'rel/')
if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' || (github.event_name == 'create' && github.event.ref_type == 'tag' && startsWith(github.event.ref, 'rel/'))
runs-on: ubuntu-latest
steps:
- name: Free disk space
Expand Down Expand Up @@ -76,8 +78,8 @@ jobs:
echo "TEZ_VERSION=${{ github.event.inputs.tezVersion }}" >> $GITHUB_ENV
echo "BUILD_ENV=release" >> $GITHUB_ENV

- name: Prepare environment variables for Release
if: github.event_name == 'create' && github.event.ref_type == 'tag' && startsWith(github.event.ref, 'rel/')
- name: Prepare environment variables from repository
if: github.event_name != 'workflow_dispatch'
run: |
echo "HIVE_VERSION=$(mvn -f "pom.xml" -q help:evaluate -Dexpression=project.version -DforceStdout)" >> $GITHUB_ENV
echo "HADOOP_VERSION=$(mvn -f "pom.xml" -q help:evaluate -Dexpression=hadoop.version -DforceStdout)" >> $GITHUB_ENV
Expand All @@ -87,10 +89,21 @@ jobs:
- name: Prepare common environment variables
run: |
echo "namespace=${{ vars.DOCKER_NAMESPACE || 'apache' }}" >> $GITHUB_ENV
echo "tag=$HIVE_VERSION" | awk '{print tolower($0)}' >> $GITHUB_ENV
echo "tag=$HIVE_VERSION" >> $GITHUB_ENV
Copy link
Copy Markdown
Contributor Author

@okumin okumin Mar 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we don't have a case where we have to keep the tolower.

Also, this PR currently assumes we want to replace 4.3.0-SNAPSHOT and standalone-metastore-4.3.0-SNAPSHOT every day. I'd say it is a good starting point; we can add the date or checksum later if we want.


- name: Validate nightly tag suffix
if: github.event_name == 'schedule'
run: |
case "$tag" in
Copy link
Copy Markdown
Member

@deniskuzZ deniskuzZ Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not simplify with if

if [[ "$tag" != *-SNAPSHOT ]]; then
  echo "Nightly image tag must end with -SNAPSHOT, got: $tag"
  exit 1
fi

*-SNAPSHOT) ;;
*)
echo "Nightly image tag must end with -SNAPSHOT, got: $tag"
exit 1
;;
esac

- name: Build Hive project
if: github.event_name == 'create' && github.event.ref_type == 'tag' && startsWith(github.event.ref, 'rel/')
if: github.event_name != 'workflow_dispatch'
run: |
mvn --batch-mode clean package -DskipTests -Pdist
ls ./packaging/target/
Expand Down
2 changes: 1 addition & 1 deletion itests/test-docker/k8s/hive-metastore/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ spec:
- |
apt-get update
apt-get install -y --no-install-recommends ca-certificates wget
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.26.19/bundle-2.26.19.jar -P /tmp/ext-jars
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.41.3/bundle-2.41.3.jar -P /tmp/ext-jars
Copy link
Copy Markdown
Contributor Author

@okumin okumin Mar 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess [this patch] brought a new library.
#6167

Should we rather exclude aws-java-sdk-secretsmanager from aws-secretsmanager-caching-java?

Copy link
Copy Markdown
Member

@deniskuzZ deniskuzZ Mar 23, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AWS SDK v2 bundle does NOT include aws-secretsmanager-caching-java, or am i missing something here?
Project is using aws-secretsmanager-caching-java:2.1.1. It already depends on AWS SDK v2, not v1.
aws-java-sdk-secretsmanager is not on the classpath

Copy link
Copy Markdown
Contributor Author

@okumin okumin Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aws-secretsmanager-caching-java 2.1.1 depends on aws-java-sdk-secretsmanager 2.41.3 and the secret manager depends on another SDK 2.41.3. The current configuration bundles both 2.41.3 from the dependency and 2.26.19 from the bundle.jar, which can cause a conflict.
As long as I surveyed this area for S3A and Credential Vending, I strongly feel hive-exec should not directly include any SDK. I am creating an addendum patch for #6167

volumeMounts:
- name: ext-jars
mountPath: /tmp/ext-jars
Expand Down
2 changes: 1 addition & 1 deletion itests/test-docker/k8s/hive/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ spec:
- |
apt-get update
apt-get install -y --no-install-recommends ca-certificates wget
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.26.19/bundle-2.26.19.jar -P /tmp/ext-jars
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.41.3/bundle-2.41.3.jar -P /tmp/ext-jars
volumeMounts:
- name: ext-jars
mountPath: /tmp/ext-jars
Expand Down
2 changes: 1 addition & 1 deletion packaging/src/docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ exit
**Disclaimer:**
Hadoop **3.4.1** requires **AWS SDK v2**.
```shell
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.26.19/bundle-2.26.19.jar -P jars/
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.41.3/bundle-2.41.3.jar -P jars/
```

2. Set the following environment variables:
Expand Down
4 changes: 2 additions & 2 deletions standalone-metastore/packaging/src/docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ change the `POSTGRES_LOCAL_PATH` to the path of the downloaded jar.
**Disclaimer:**
Hadoop **3.4.1** requires **AWS SDK v2**.
```shell
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.26.19/bundle-2.26.19.jar -P jars/
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.41.3/bundle-2.41.3.jar -P jars/
```

2. Set the following environment variables:
Expand Down Expand Up @@ -168,4 +168,4 @@ Volumes:
To stop/remove them all,
```shell
docker compose down
```
```
Loading