Skip to content

Parallelise our CI/CD jobs #250

@CMCDragonkai

Description

@CMCDragonkai

Details from https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/184#note_668294739

Using an ever longer delay is not a good way of doing these tests, they don't scale. Ideally we fix this up before going into prod.

I also noticed our tests taking longer now. Up to 30 minutes.

To speed up our testing, we can parallelise our testing. Now our jest tests are naturally parallelised by checking the number of cores the host environment has. But the CI/CD runners all have 1 CPU core:

If you’re using the GitLab.com 32 autoscaling shared runners, I believe the runners have 1CPU, 4GB RAM, and 22Gb of storage (16Gb available).

There should be a way for us to configure with more cores or other wise increase the number of parallel jobs by splitting the test job, which would end up using multiple runners (somewhat inefficient) due to duplication of resources, and then adding our own custom runner with more cores.

A quick way to do this would to be create separate CI/CD jobs for each domain run. So npm test tests/X and npm test tests/Y might allow us to take advantage of gitlab CI/CD's splitting of each job in parallel. Only issue is the sharing of the Nix environment.

Tasks

  1. - Investigate dynamic parent child pipelines
  2. - Test splitting can be automatic by splitting a pipeline for each subdirectory under tests/*
  3. - For all the immediate files in tests/* they can run under a single test pipeline

Metadata

Metadata

Assignees

No one assigned

    Labels

    procedureAction that must be executed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions