Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ repos:
- id: terraform-docs-go
args: ["markdown", "table", "--config", "./.terraform-docs.yaml", "--recursive", "--output-file", "README.md", "./"]
- repo: https://github.com/bridgecrewio/checkov.git
rev: '3.2.461'
rev: '3.2.471'
hooks:
- id: checkov
verbose: false
30 changes: 16 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ For each new ECR repository (or linked collection of ECR repositories), create a
* a stage_build caller workflow
* a prod_promote caller workflow

The [ppod_ecr.tf](./ppod_ecr.tf) is a good example of a single ECR repository for a Lambda function build around a containerized app. The [timdex_ecrs.tf](./timdex_ecrs.tf) is a good example of a collection of ECR repositories all linked to one project (and there are both Fargate-linked ECRs and Lambda-linked ECRs in that file).
The [ppod_ecr.tf](./ppod_ecr.tf) is a good example of a single ECR repository for a Lambda function built around a containerized app. The [timdex_ecrs.tf](./timdex_ecrs.tf) is a good example of a collection of ECR repositories all linked to one project (and there are both Fargate-linked ECRs and Lambda-linked ECRs in that file).

**Note**: For Lambda function ECRs, it is imperative that the Infra engineer coordinates with the software engineer to determine the name of the Lambda function as part of the creation of the ECR by this repository.

Expand Down Expand Up @@ -96,38 +96,40 @@ This is a core infrastructure repository that defines infrastructure related to
* [Alma Hook](https://github.com/MITLibraries/mitlib-tf-workloads-almahook)
* [Alma Webhook Lambdas](https://github.com/MITLibraries/alma-webhook-lambdas)
* [Alma Patron Load](https://github.com/MITLibraries/mitlib-tf-workloads-patronload)
* [Alma Patron Load Application Container](https://github.com/MITLibraries/alma-patronload)
* [Alma Patron Load Application](https://github.com/MITLibraries/alma-patronload)
* [Archival Packaging Tool](https://github.com/MITLibraries/mitlib-tf-workloads-apt)
* [Archival Packaging Tool Application Container](https://github.com/MITLibraries/archival-packaging-tool)
* [Archival Packaging Tool Application](https://github.com/MITLibraries/archival-packaging-tool)
* [ASATI](https://github.com/MITLibraries/mitlib-tf-workloads-asati)
* [ASATI Application Container](https://github.com/MITLibraries/asati)
* [ASATI Application](https://github.com/MITLibraries/asati)
* [Carbon](https://github.com/MITLibraries/mitlib-tf-workloads-carbon)
* [CDPS](https://github.com/MITLibraries/mitlib-tf-workloads-cdps-storage)
* [S3 BagIt Validator](https://github.com/MITLibraries/s3-bagit-validator)
* [CDPS CURT](https://github.com/MITLibraries/cdps-curt)
* [DSC](https://github.com/MITLibraries/mitlib-tf-workloads-dsc)
* [DSC Application Container](https://github.com/MITLibraries/dspace-submission-composer)
* [DSS](https://github.com/MITLibraries/mitlib-tf-workloads-dss)
* [DSpace Submission Service Application Container](https://github.com/MITLibraries/dspace-submission-service)
* [ETD](https://github.com/MITLibraries/mitlib-tf-workloads-etd)
* [DSO Infrastructure](https://github.com/MITLibraries/mitlib-tf-workloads-dso)
* [DSpace Submission Composer Application](https://github.com/MITLibraries/dspace-submission-composer)
* [DSpace Submission Service Application](https://github.com/MITLibraries/dspace-submission-service)
* **DEPRECATED**: [DSC](https://github.com/MITLibraries/mitlib-tf-workloads-dsc)
* **DEPRECATED**: [DSS](https://github.com/MITLibraries/mitlib-tf-workloads-dss)
* [ETD Infrastructure](https://github.com/MITLibraries/mitlib-tf-workloads-etd)
* [HRQB](https://github.com/MITLibraries/mitlib-tf-workloads-hrqb-loader)
* [HRQB Client](https://github.com/MITLibraries/hrqb-client)
* [marimo notebooks](https://github.com/MITLibraries/mitlib-tf-workloads-notebooks)
* [marimo-launcher](https://github.com/MITLibraries/marimo-launcher)
* [marimo-helloworld](https://github.com/MITLibraries/marimo-helloworld)
* [Matomo](https://github.com/MITLibraries/mitlib-tf-workloads-matomo)
* [Matomo Application Container](https://github.com/MITLibraries/docker-matomo)
* [Matomo Application](https://github.com/MITLibraries/docker-matomo)
* [PPOD](https://github.com/MITLibraries/mitlib-tf-workloads-ppod)
* [PPOD Application Container](https://github.com/MITLibraries/ppod)
* [PPOD Application](https://github.com/MITLibraries/ppod)
* [TACOS](https://github.com/MITLibraries/mitlib-tf-workloads-tacos)
* [tacos-detectors-lambdas](https://github.com/MITLibraries/tacos-detectors-lambdas)
* [TIMDEX](https://github.com/MITLibraries/mitlib-tf-workloads-timdex-infrastructure)
* [TIMDEX Application Container](https://github.com/MITLibraries/timdex)
* [TIMDEX Application](https://github.com/MITLibraries/timdex)
* [TIMDEX Dataset API](https://github.com/MITLibraries/timdex-dataset-api)
* [TIMDEX Index Manager](https://github.com/MITLibraries/timdex-index-manager)
* [TIMDEX Pipeline Lambdas](https://github.com/MITLibraries/timdex-pipeline-lambdas)
* [TIMDEX UI](https://github.com/MITLibraries/timdex-ui)
* [TIMDEX Simulator](https://github.com/MITLibraries/timdex-simulator)
* [TIMDEX Transmogrifier](https://github.com/MITLibraries/transmogrifier)
* [WCD2Reshare](https://github.com/MITLibraries/mitlib-tf-workloads-wcd2reshare)
* [WCD2Reshare Application Container](https://github.com/MITLibraries/wcd2reshare)
* **DEPRECATED**: [Wiley](https://github.com/MITLibraries/mitlib-tf-workloads-wiley)
Expand All @@ -137,7 +139,7 @@ This is a core infrastructure repository that defines infrastructure related to

* Owner: See [CODEOWNERS](./.github/CODEOWNERS)
* Team: See [CODEOWNERS](./.github/CODEOWNERS)
* Last Maintenance: 2025-08
* Last Maintenance: 2025-09

## TF markdown is automatically inserted at the bottom of this file, nothing should be written beyond this point

Expand All @@ -146,7 +148,7 @@ This is a core infrastructure repository that defines infrastructure related to

| Name | Version |
|------|---------|
| terraform | ~> 1.11 |
| terraform | ~> 1.13 |
| aws | ~> 5.0 |

## Providers
Expand Down
8 changes: 4 additions & 4 deletions ect_workflow_text_ecr.tf → ecr_workflow_test_ecr.tf
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ module "ecr_workflowtest" {
## For workflowtest application repo and ECR repository
# Outputs in dev
output "workflowtest_dev_build_workflow" {
value = var.environment == "prod" || var.environment == "stage" ? null : templatefile("${path.module}/files/dev-build.tpl", {
value = var.environment == "prod" || var.environment == "stage" ? null : templatefile("${path.module}/files/dev-build-cpu-arch.tpl", {
region = var.aws_region
role = module.ecr_workflowtest.gha_role
ecr = module.ecr_workflowtest.repository_name
Expand All @@ -32,7 +32,7 @@ output "workflowtest_dev_build_workflow" {
description = "Full contents of the dev-build.yml for the ecr-workflow-test repo"
}
output "workflowtest_makefile" {
value = var.environment == "prod" || var.environment == "stage" ? null : templatefile("${path.module}/files/makefile.tpl", {
value = var.environment == "prod" || var.environment == "stage" ? null : templatefile("${path.module}/files/makefile-cpu-arch.tpl", {
ecr_name = module.ecr_workflowtest.repository_name
ecr_url = module.ecr_workflowtest.repository_url
function = ""
Expand All @@ -43,7 +43,7 @@ output "workflowtest_makefile" {

# Outputs in stage
output "workflowtest_stage_build_workflow" {
value = var.environment == "prod" || var.environment == "dev" ? null : templatefile("${path.module}/files/stage-build.tpl", {
value = var.environment == "prod" || var.environment == "dev" ? null : templatefile("${path.module}/files/stage-build-cpu-arch.tpl", {
region = var.aws_region
role = module.ecr_workflowtest.gha_role
ecr = module.ecr_workflowtest.repository_name
Expand All @@ -55,7 +55,7 @@ output "workflowtest_stage_build_workflow" {

# Outputs after promotion to prod
output "workflowtest_prod_promote_workflow" {
value = var.environment == "stage" || var.environment == "dev" ? null : templatefile("${path.module}/files/prod-promote.tpl", {
value = var.environment == "stage" || var.environment == "dev" ? null : templatefile("${path.module}/files/prod-promote-cpu-arch.tpl", {
region = var.aws_region
role_stage = "${module.ecr_workflowtest.repo_name}-gha-stage"
role_prod = "${module.ecr_workflowtest.repo_name}-gha-prod"
Expand Down
17 changes: 17 additions & 0 deletions files/dev-build-cpu-arch-extra-region.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
### This is the Terraform-generated extra workflow job for the ###
### ${ecr} app repository. ###
### This should be added to jobs section of the dev-build.yml. If this is ###
### a Lambda function, uncomment the FUNCTION: line ###

deploy-${region}:
needs: prep
name: Dev Deploy ${region}
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-dev.yml@main
secrets: inherit
with:
AWS_REGION: "${region}"
GHA_ROLE: "${role}"
ECR: "${ecr}"
CPU_ARCH: $${{ needs.prep.outputs.cpuarch }}
# FUNCTION: "${function}"
# PREBUILD:
60 changes: 60 additions & 0 deletions files/dev-build-cpu-arch.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
### This is the Terraform-generated dev-build.yml workflow for the ###
### ${ecr} app repository. ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
### the document. If the container requires any additional pre-build ###
### commands, uncomment and edit the PREBUILD line at the end of the ###
### document. ###

name: Dev Container Build and Deploy
on:
workflow_dispatch:
pull_request:
branches:
- main
paths-ignore:
- '.github/**'

permissions:
id-token: write
contents: read

jobs:
prep:
name: Prep for Build
runs-on: ubuntu-latest
outputs:
cpuarch: $${{ steps.setarch.outputs.cpuarch }}
steps:
- name: Checkout
uses: actions/checkout@v5

- name: Set CPU Architecture
id: setarch
run: |
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
if [[ -f .aws-architecture ]]; then
ARCH=$(cat .aws-architecture)
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
else
ARCH="linux/amd64"
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
fi
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
echo "$ARCH is INVALID architecture!"
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
exit 1
fi
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT

deploy:
needs: prep
name: Dev Deploy
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-dev.yml@main
secrets: inherit
with:
AWS_REGION: "${region}"
GHA_ROLE: "${role}"
ECR: "${ecr}"
CPU_ARCH: $${{ needs.prep.outputs.cpuarch }}
# FUNCTION: "${function}"
# PREBUILD:
57 changes: 57 additions & 0 deletions files/makefile-cpu-arch.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
### This is the Terraform-generated header for ${ecr_name}. If ###
### this is a Lambda repo, uncomment the FUNCTION line below ###
### and review the other commented lines in the document. ###
ECR_NAME_DEV := ${ecr_name}
ECR_URL_DEV := ${ecr_url}
CPU_ARCH ?= $(shell cat .aws-architecture 2>/dev/null || echo "linux/amd64")
# FUNCTION_DEV := ${function}
### End of Terraform-generated header ###


### Terraform-generated Developer Deploy Commands for Dev environment ###
check-arch:
@ARCH_FILE=".aws-architecture"; \
if [[ "$(CPU_ARCH)" != "linux/amd64" && "$(CPU_ARCH)" != "linux/arm64" ]]; then \
echo "Invalid CPU_ARCH: $(CPU_ARCH)"; exit 1; \
fi; \
if [[ -f $$ARCH_FILE ]]; then \
echo "latest-$(shell echo $(CPU_ARCH) | cut -d'/' -f2)" > .arch_tag; \
else \
echo "latest" > .arch_tag; \
fi

dist-dev: check-arch ## Build docker container (intended for developer-based manual build)
@ARCH_TAG=$$(cat .arch_tag); \
docker buildx inspect $(ECR_NAME_DEV) >/dev/null 2>&1 || docker buildx create --name $(ECR_NAME_DEV) --use; \
docker buildx use $(ECR_NAME_DEV); \
docker buildx build --platform $(CPU_ARCH) \
--load \
--tag $(ECR_URL_DEV):make-$$ARCH_TAG \
--tag $(ECR_URL_DEV):make-$(shell git describe --always) \
--tag $(ECR_NAME_DEV):$$ARCH_TAG \
.

publish-dev: dist-dev ## Build, tag and push (intended for developer-based manual publish)
@ARCH_TAG=$$(cat .arch_tag); \
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $(ECR_URL_DEV); \
docker push $(ECR_URL_DEV):make-$$ARCH_TAG; \
docker push $(ECR_URL_DEV):make-$(shell git describe --always); \
docker push $(ECR_URL_DEV):make-$(shell echo $(CPU_ARCH) | cut -d'/' -f2)

### If this is a Lambda repo, uncomment the two lines below ###
# update-lambda-dev: ## Updates the lambda with whatever is the most recent image in the ecr (intended for developer-based manual update)
# @ARCH_TAG=$$(cat .arch_tag); \
# aws lambda update-function-code \
# --region us-east-1 \
# --function-name $(FUNCTION_DEV) \
# --image-uri $(ECR_URL_DEV):make-$$ARCH_TAG

docker-clean: ## Clean up Docker detritus
@ARCH_TAG=$$(cat .arch_tag); \
echo "Cleaning up Docker leftovers (containers, images, builders)"; \
docker rmi -f $(ECR_URL_DEV):make-$$ARCH_TAG; \
docker rmi -f $(ECR_URL_DEV):make-$(shell git describe --always) || true; \
docker rmi -f $(ECR_URL_DEV):make-$(shell echo $(CPU_ARCH) | cut -d'/' -f2) || true; \
docker rmi -f $(ECR_NAME_DEV):$$ARCH_TAG || true; \
docker buildx rm $(ECR_NAME_DEV) || true
@rm -rf .arch_tag
17 changes: 17 additions & 0 deletions files/prod-promote-cpu-arch-extra-region.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
### This should be added to jobs section of the prod-promote.yml.
### If this is a Lambda function, uncomment the FUNCTION: line

deploy-${region}:
needs: prep
name: Deploy ${region}
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-promote-prod.yml@main
secrets: inherit
with:
AWS_REGION: "${region}"
GHA_ROLE_STAGE: ${role_stage}
GHA_ROLE_PROD: ${role_prod}
ECR_STAGE: "${ecr_stage}"
ECR_PROD: "${ecr_prod}"
CPU_ARCH: $${{ needs.prep.outputs.cpuarch }}
# FUNCTION: "${function}"

57 changes: 57 additions & 0 deletions files/prod-promote-cpu-arch.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
### This is the Terraform-generated prod-promote.yml workflow for the ###
### ${ecr_prod} repository. ###
### If this is a Lambda repo, uncomment the FUNCTION line at the end of ###
### the document. ###

name: Prod Container Promote
on:
workflow_dispatch:
release:
types: [published]

permissions:
id-token: write
contents: read

jobs:
prep:
name: Prep for Promote
runs-on: ubuntu-latest
outputs:
cpuarch: $${{ steps.setarch.outputs.cpuarch }}
steps:
- name: Checkout
uses: actions/checkout@v5

- name: Set CPU Architecture
id: setarch
run: |
echo "### :abacus: Architecture Selection" >> $GITHUB_STEP_SUMMARY
if [[ -f .aws-architecture ]]; then
ARCH=$(cat .aws-architecture)
echo "\`$ARCH\` was read from \`.aws-architecture\` and passed to the deploy job." >> $GITHUB_STEP_SUMMARY
else
ARCH="linux/amd64"
echo "No \`.aws-architecture\` file, so default \`$ARCH\` was passed to the deploy job." >> $GITHUB_STEP_SUMMARY
fi
if [[ "$ARCH" != "linux/arm64" && "$ARCH" != "linux/amd64" ]]; then
echo "$ARCH is INVALID architecture!"
echo "$ARCH is INVALID architecture!" >> $GITHUB_STEP_SUMMARY
exit 1
fi
echo "cpuarch=$ARCH" >> $GITHUB_OUTPUT

deploy:
needs: prep
name: Deploy
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-promote-prod.yml@main
secrets: inherit
with:
AWS_REGION: "${region}"
GHA_ROLE_STAGE: ${role_stage}
GHA_ROLE_PROD: ${role_prod}
ECR_STAGE: "${ecr_stage}"
ECR_PROD: "${ecr_prod}"
CPU_ARCH: $${{ needs.prep.outputs.cpuarch }}
# FUNCTION: "${function}"

17 changes: 17 additions & 0 deletions files/stage-build-cpu-arch-extra-region.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
### This is the Terraform-generated extra workflow job for the ###
### ${ecr} app repository. ###
### This should be added to jobs section of the stage-build.yml. If this ###
### is a Lambda function, uncomment the FUNCTION: line ###

deploy-${region}:
needs: prep
name: Stage Deploy ${region}
uses: mitlibraries/.github/.github/workflows/ecr-multi-arch-deploy-stage.yml@main
secrets: inherit
with:
AWS_REGION: "${region}"
GHA_ROLE: "${role}"
ECR: "${ecr}"
CPU_ARCH: $${{ needs.prep.outputs.cpuarch }}
# FUNCTION: "${function}"
# PREBUILD:
Loading