Only keep 10 images at a time in ECR

[kalilsn]

Issue(s) Resolved

We currently have over 7000 images in each of our repositories, because we have no lifecycle policy.

NOTE:
Currently this plan is going to delete the sitebuilder repository.

Plan output for global-aws env (not stevie)
Terraform will perform the following actions:

  # module.ecr_repositories.aws_ecr_lifecycle_policy.nginx will be created
  + resource "aws_ecr_lifecycle_policy" "nginx" {
      + id          = (known after apply)
      + policy      = jsonencode(
            {
              + rules = [
                  + {
                      + action       = {
                          + type = "expire"
                        }
                      + description  = "Only keep 10 images at a time"
                      + rulePriority = 1
                      + selection    = {
                          + countNumber = 10
                          + countType   = "imageCountMoreThan"
                          + tagStatus   = "any"
                        }
                    },
                ]
            }
        )
      + registry_id = (known after apply)
      + repository  = "nginx"
    }

  # module.ecr_repositories.aws_ecr_lifecycle_policy.pubpub_v7 will be created
  + resource "aws_ecr_lifecycle_policy" "pubpub_v7" {
      + id          = (known after apply)
      + policy      = jsonencode(
            {
              + rules = [
                  + {
                      + action       = {
                          + type = "expire"
                        }
                      + description  = "Only keep 10 images at a time"
                      + rulePriority = 1
                      + selection    = {
                          + countNumber = 10
                          + countType   = "imageCountMoreThan"
                          + tagStatus   = "any"
                        }
                    },
                ]
            }
        )
      + registry_id = (known after apply)
      + repository  = "pubpub-v7"
    }

  # module.ecr_repositories.aws_ecr_lifecycle_policy.pubpub_v7_core will be created
  + resource "aws_ecr_lifecycle_policy" "pubpub_v7_core" {
      + id          = (known after apply)
      + policy      = jsonencode(
            {
              + rules = [
                  + {
                      + action       = {
                          + type = "expire"
                        }
                      + description  = "Only keep 10 images at a time"
                      + rulePriority = 1
                      + selection    = {
                          + countNumber = 10
                          + countType   = "imageCountMoreThan"
                          + tagStatus   = "any"
                        }
                    },
                ]
            }
        )
      + registry_id = (known after apply)
      + repository  = "pubpub-v7-core"
    }

  # module.ecr_repositories.aws_ecr_lifecycle_policy.pubpub_v7_jobs will be created
  + resource "aws_ecr_lifecycle_policy" "pubpub_v7_jobs" {
      + id          = (known after apply)
      + policy      = jsonencode(
            {
              + rules = [
                  + {
                      + action       = {
                          + type = "expire"
                        }
                      + description  = "Only keep 10 images at a time"
                      + rulePriority = 1
                      + selection    = {
                          + countNumber = 10
                          + countType   = "imageCountMoreThan"
                          + tagStatus   = "any"
                        }
                    },
                ]
            }
        )
      + registry_id = (known after apply)
      + repository  = "pubpub-v7-jobs"
    }

  # module.ecr_repositories.aws_ecr_repository.pubpub_v7_site_builder will be destroyed
  # (because aws_ecr_repository.pubpub_v7_site_builder is not in configuration)
  - resource "aws_ecr_repository" "pubpub_v7_site_builder" {
      - arn                  = "arn:aws:ecr:us-east-1:246372085946:repository/pubpub-v7-site-builder" -> null
      - id                   = "pubpub-v7-site-builder" -> null
      - image_tag_mutability = "MUTABLE" -> null
      - name                 = "pubpub-v7-site-builder" -> null
      - registry_id          = "246372085946" -> null
      - repository_url       = "246372085946.dkr.ecr.us-east-1.amazonaws.com/pubpub-v7-site-builder" -> null
      - tags                 = {} -> null
      - tags_all             = {} -> null

      - encryption_configuration {
          - encryption_type = "AES256" -> null
        }

      - image_scanning_configuration {
          - scan_on_push = false -> null
        }
    }

Plan: 4 to add, 0 to change, 1 to destroy.

[tefkah]

i think the idea is pretty good (we are paying more and more each month for nothing), and destroying the site-builder repo is not a big deal (i can just re-add it).

scenario im worried about:

• we deploy main to stevie
• we push 10 individual commits to other prs, which create 10 new images
• we either want to redeploy stevie, or scale it up ???

does this not cause issues in that case? we could ofc instead deploy off ghcr:latest, or create some kind of :latest tag in ECR that we preserve, or maybe this scenario wouldn't cause any problems?

[kalilsn]

i think the idea is pretty good (we are paying more and more each month for nothing), and destroying the site-builder repo is not a big deal (i can just re-add it).

scenario im worried about:

• we deploy main to stevie
• we push 10 individual commits to other prs, which create 10 new images
• we either want to redeploy stevie, or scale it up ???

does this not cause issues in that case? we could ofc instead deploy off ghcr:latest, or create some kind of :latest tag in ECR that we preserve, or maybe this scenario wouldn't cause any problems?

Good question, and probably correct. I forgot that we were putting individual commit images in the same repository. We should tag the actually deployed releases and then I can update this policy to take that into account.

[kalilsn]

Or yeah, we could just deploy directly from ghcr and stop spending all this money on ECR entirely

[tefkah]

yeah i think that is likely fine, i don't see much benefit anymore from using two registries here. the only ones i can think of are

1. we somehow don't want people to access the "in-between" images (not really a concern)
2. pull speed is maybe faster for (preview) deployments from ECR for AWS services than from GHCR. not really a big deal id say