Split E2E Test tasks into multiple subtasks#1811
Conversation
Mythicaeda
added
ci
dandelany
left a comment
dandelany
left a comment
There was a problem hiding this comment.
A few small comments below from reviewing with @Mythicaeda
dandelany
left a comment
dandelany
left a comment
There was a problem hiding this comment.
@Mythicaeda I pushed two more commits to this branch for your review:
- 8a773ed is explained in this comment
- 9d4eaac changes the node cache used for gradle builds to be more aligned with what
setup-nodedoes, which is to use the global~/.npmcache rather than caching individualnode_modulesfolders which are more fragile/disposable. In particular,npm cithrows away the node_modules folder but does make use of the global cache. Even though our services use different Node versions, it's OK for them to share this cache! The setup-node docs mention this - "The action follows actions/cache guidelines, and caches global cache on the machine instead of node_modules, so cache can be reused between different Node.js versions."
Let me know if you have any concerns - if not this is 👍 to merge.
|
The one thing I'd like to confirm before merging is that sequencing server and action server are able to use the |
Yeah, doesn't seem like it's possible in the Github UI until some version of the workflow has been merged to main :-/ Feel free to push a temp commit or you may also be able to trigger it via a GH API call |
Upon further examination, all Action Server tests are currently unit tests that can be run with the entire system down, not E2E tests
Java currently takes multiple minutes to compile. This is more than sufficient time for Hasura to complete its start up tasks, instead of needing to delay for 30s after starting up.
- using `false` for this test run, but it should be configured to compare to develop (defaults to only caching `main` and `master`, which aren't branches in the repo)
…r" step - Setup Gradle includes the `validate wrapper` step - activate gradle caching during `setup gradle`
…H workflow cache keys
|
I updated the required checks on the |
2 participants
Description
Currently, the E2E Test task takes 15 minutes to execute, which is absurdly high. This PR aims to improve that number without making the tests flaky.
Breakdown of E2E Suite test speed locally (with services already setup):
:e2eTest:e2eTest:: ~6 mins:dbTest:e2eTest:: ~2 mins:sequencing-server:e2eTest:: ~1 min:action-server:e2eTest:: ~5 secThe first approach is to split the tests by suite into multiple parallel jobs. The suites should already be independent (especially as they're a mixture of Java and TS tests), but by running them in separate jobs we avoid any issues where two suites are trying to alter the DB at the same time. Additionally, by having them as separate jobs, it will be clearer which suite in particular is failing, as opposed to needing to go through the collected folder.
Breakdown of tests after this approach:
unitTests: 6 min (2 min running tests, 3.5 min assembling):e2eTest:e2eTest:: 11 min (5 min running tests, 3.5 min assembling):dbTest:e2eTest:: 4.5 min (4 min running tests):sequencing-server:e2eTest:: 7 min (1.5 min running tests, 3.5 min assembling the Java backend):action-server:e2eTest:: 5 min (10s running tests, 3 min assembling the Java backend)After that, I added the following improvements
e2e-testenvironment, it doesn't actually require the backend to be runningdevelopnode_modulesforaction-serverandsequencing-server. These seemed to not be cached by the gradle caching task, so they're now being manually cachedBreakdown of tests after all of this:
unitTests: 4.5 min (2 min running tests, 2 min assembling):e2eTest:e2eTest:: 9 min (5 min running tests, 2 min assembling):dbTest:e2eTest:: 4.5 min (4 min running tests):sequencing-server:e2eTest:: 5 min (1.5 min running tests, 2 min assembling):action-server:e2eTest:: 30s min (10s running tests)