Skip to content

Add GHCR-seeded PostgreSQL images and wire CI matrix jobs to consume deterministic pgseed tags#164

Closed
Copilot wants to merge 10 commits into
mainfrom
copilot/add-seeded-postgres-images
Closed

Add GHCR-seeded PostgreSQL images and wire CI matrix jobs to consume deterministic pgseed tags#164
Copilot wants to merge 10 commits into
mainfrom
copilot/add-seeded-postgres-images

Conversation

Copy link
Copy Markdown

Copilot AI commented May 7, 2026

  • Re-review current phpunit restore/upgrade flow and pgseed image behavior
  • Remove util.php --restore usage from CI phpunit flow
  • Add docker-level sitedata restore path by copying packaged sitedata from the pgseed postgres service container
  • Ensure pgseed image exposes a deterministic sitedata archive path in-container
  • Update docs to describe docker-level sitedata restore behavior
  • Re-run targeted syntax validation and parallel validation
Original prompt

Create a PR in catalyst/catalyst-moodle-workflows (base branch main) implementing Option A: build and publish seeded PostgreSQL images to GHCR and use them in CI.

Context:

  • Current CI workflow uses services.postgres.image: postgres:${{ matrix.pgsql-ver }} in .github/workflows/ci.yml for phpunit and behat jobs.
  • We want to speed up CI by pulling a pre-seeded PostgreSQL image (pre-initialized unit test DB) instead of creating it from scratch.
  • A separate change in Moodle core will allow unit test DB to be upgraded; assume that is already working.

Requirements:

  1. Add a new GitHub Actions workflow (e.g. .github/workflows/build-pgseed.yml) that builds and pushes seeded Postgres images to GHCR.

    • Build a matrix based on existing .github/actions/matrix/matrix_includes.yml (similar to .github/workflows/build-dockerfiles.yml).
    • Focus on PostgreSQL only.
    • Tag scheme MUST include:
      • moodle branch (e.g. MOODLE_403_STABLE)
      • PHP version (if relevant to seeding; include if needed)
      • PostgreSQL version (e.g. 16)
      • Moodle minor version which is a 10-digit number (exact source to be determined from repo’s existing matrix parsing logic; use existing scripts/actions if available to obtain it reliably).
    • Publish to GHCR under ghcr.io/catalyst/... with deterministic tag(s).
    • Prefer immutable tags (include all components) and optionally also push a latest tag per tuple.
    • Use docker/login-action@v3 with GITHUB_TOKEN and set job permissions to allow pushing packages.
    • Avoid rebuilding if nothing relevant changed when possible (e.g., only run on changes in seed Dockerfile/scripts/matrix file, plus workflow_dispatch; optionally scheduled).
  2. Create the Docker build context necessary for the seeded image.

    • Add a Dockerfile and scripts under docker/pgseed/ (or similar) that produce a Postgres image with the seeded DB.
    • The implementation should be designed to work with the Moodle core change that enables upgrading a unit test DB.
    • Keep image size reasonable; document assumptions.
  3. Update .github/workflows/ci.yml to use the seeded Postgres image for jobs that currently use Postgres service.

    • At minimum: phpunit job’s services.postgres.image should reference the seeded image tag computed from the matrix fields including the 10-digit moodle minor version.
    • Optionally update behat/validate/mustache/grunt jobs similarly if safe; make it configurable.
  4. Update matrix generation if needed to pass through the moodle minor version and/or pg seed image reference.

    • Prefer to extend existing parsing (parse-version action/script) to output the minor version and include it in phpunit/behat matrices.
  5. Add documentation:

    • Brief notes in README or docs describing how the seeded images are built, tags, and how to regenerate.

Deliverables:

  • PR with all code changes.
  • Ensure workflows are syntactically correct.

Repository details:

  • Repo: catalyst/catalyst-moodle-workflows
  • Base branch: main

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: what is the fastest way to restore a database into a specific shape to make github or gitlab ci faster

Assistant: The fastest way is to stop restoring full backups on every CI run and instead start each job from a pre-shaped database snapshot/image (or a template DB) that already contains the schema + baseline data you need. Then apply only a tiny delta (migrations for that commit, or a small fixture set). Which mechanism is “fastest” depends on your database engine:

Best options (fastest → slower)

1) Use a pre-baked DB image/snapshot (CI pulls it)

When it’s fastest: integration tests need realistic data and you can keep a “golden” dataset.

  • Postgres: build a Docker image that already contains the data directory, or ship a compressed pg_basebackup/pg_dump artifact and restore from local cache.
  • MySQL: same idea—pre-baked datadir in an image or a prepared backup artifact.
  • CI benefit: startup becomes “docker pull + container start”, often seconds to ~1 minute instead of many minutes.

Key trick: have a scheduled pipeline (nightly or on-demand) that regenerates this snapshot when baseline data changes, and publish it to your registry/object storage.

2) Postgres: use a TEMPLATE database + CREATE DATABASE … TEMPLATE …

Often the best “pure Postgres” solution if your tests can share a server per job.

  • Prepare template_ci once (schema + baseline fixtures).
  • For each test run: CREATE DATABASE test_db TEMPLATE template_ci;
  • This is basically a filesystem-level copy inside Postgres and is extremely fast.

Constraints:

  • Template DB must have no active connections.
  • Works best when the CI job has its own Postgres instance (service container) so parallel jobs don’t fight.

3) Copy-on-write storage snapshots (ZFS/LVM/EBS) if you control runners

If you run **self-hosted ru...

This pull request was created from Copilot chat.

Copilot AI changed the title [WIP] Add seeded PostgreSQL images for faster CI Add GHCR-seeded PostgreSQL images and wire CI matrix jobs to consume deterministic pgseed tags May 7, 2026
Copilot AI requested a review from brendanheywood May 7, 2026 17:27
Copilot AI and others added 6 commits May 7, 2026 17:34
@jay-oswald
Copy link
Copy Markdown
Contributor

This has been re-done in #166

@jay-oswald jay-oswald closed this May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants