Run-workers: Parallelization per suite doesn't work in specific case in 3.7.6

[mirao]

@kobenguyent Good point! Definitely it's caused by #5386. If I comment out this.testFiles.sort() in lib/codecept.js, it works well. CC: @DenysKuchma

Can't reproduce in my test project (maybe I need to have some more test suites with various names...), but I'm able to reproduce it in my "production" project where many test suites already are.

With test sorting, the parallelization per suite doesn't work and the test is slow 🐞 :

mirao@rog:~/workspace/codeceptjs/tests/s4e$ time codeceptjs run-workers 3 dev --by suite --grep "ClassPreparation|CourseDetails"
CodeceptJS v3.7.6 #StandWithUkraine
Running tests in 3 workers...

[Worker 04]   ✔ ClassPreparation @group1 › Get an admin token in 255ms
[Worker 05]   ✔ ClassPreparation @group1 › Get an admin token in 243ms
[Worker 04]   ✔ ClassPreparation @group1 › Create a teacher in 682ms
[Worker 05]   ✔ ClassPreparation @group1 › Create a teacher in 675ms
[Worker 06]   ✔ ClassPreparation @group1 › Get an admin token in 269ms
[Worker 06]   ✔ ClassPreparation @group1 › Create a teacher in 682ms
[Worker 04]   ✔ ClassPreparation @group1 › Prepare a school, school course, section in 2084ms
[Worker 05]   ✔ ClassPreparation @group1 › Prepare a school, school course, section in 2068ms
[Worker 05]   ✔ ClassPreparation @group1 › Create a schedule for the section in 195ms
[Worker 04]   ✔ ClassPreparation @group1 › Create a schedule for the section in 266ms
[Worker 05]   ✔ ClassPreparation @group1 › Create a school calendar in 542ms
[Worker 04]   ✔ ClassPreparation @group1 › Create a school calendar in 533ms
[Worker 06]   ✔ ClassPreparation @group1 › Prepare a school, school course, section in 2084ms
[Worker 06]   ✔ ClassPreparation @group1 › Create a schedule for the section in 190ms
[Worker 06]   ✔ ClassPreparation @group1 › Create a school calendar in 535ms
[Worker 04]   ✔ ClassPreparation @group1 › S4W: Login in 13430ms
[Worker 05]   ✔ ClassPreparation @group1 › S4W: Login in 14631ms
[Worker 06]   ✔ ClassPreparation @group1 › S4W: Login in 14726ms
[Worker 04]   ✔ ClassPreparation @group1 › S4W: Create an autonomous iPlanner in 3107ms
[Worker 05]   ✔ ClassPreparation @group1 › S4W: Create an autonomous iPlanner in 3318ms
[Worker 06]   ✔ ClassPreparation @group1 › S4W: Create an autonomous iPlanner in 3227ms
[Worker 04]   ✔ ClassPreparation @group1 › S4W: Create a lesson unit and a lesson in 3352ms
[Worker 05]   ✔ ClassPreparation @group1 › S4W: Create a lesson unit and a lesson in 3616ms
[Worker 06]   ✔ ClassPreparation @group1 › S4W: Create a lesson unit and a lesson in 3503ms
[Worker 04]   ✔ ClassPreparation @group1 › S4W: Add an instrument and set it for class preparation in 6706ms
[Worker 04]   ✔ ClassPreparation @group1 › Assign the iPlanner to the section in 1ms
[Worker 05]   ✔ ClassPreparation @group1 › S4W: Add an instrument and set it for class preparation in 6966ms
[Worker 05]   ✔ ClassPreparation @group1 › Assign the iPlanner to the section in 364ms
[Worker 06]   ✔ ClassPreparation @group1 › S4W: Add an instrument and set it for class preparation in 6931ms
[Worker 06]   ✔ ClassPreparation @group1 › Assign the iPlanner to the section in 363ms
[Worker 04]   ✔ ClassPreparation @group1 › S4E: Login as a teacher in 5963ms
[Worker 05]   ✔ ClassPreparation @group1 › S4E: Login as a teacher in 6423ms
[Worker 04]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default class preparation view in 2656ms
[Worker 06]   ✔ ClassPreparation @group1 › S4E: Login as a teacher in 6623ms
[Worker 04]   ✔ ClassPreparation @group1 › S4E: Start a class with the lesson in 2107ms
[Worker 05]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default class preparation view in 2701ms
[Worker 06]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default class preparation view in 2748ms
[Worker 04]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default teacher view in 2151ms
[Worker 05]   ✔ ClassPreparation @group1 › S4E: Start a class with the lesson in 2160ms
[Worker 04]   ✔ ClassPreparation @group1 › S4W: Disable the instrument for class preparation in 1637ms
[Worker 06]   ✔ ClassPreparation @group1 › S4E: Start a class with the lesson in 2215ms
[Worker 05]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default teacher view in 2151ms
[Worker 05]   ✔ ClassPreparation @group1 › S4W: Disable the instrument for class preparation in 1657ms
[Worker 06]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default teacher view in 2234ms
[Worker 04]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has missing class preparation view in 3283ms
[Worker 06]   ✔ ClassPreparation @group1 › S4W: Disable the instrument for class preparation in 1690ms
[Worker 05]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has missing class preparation view in 3242ms
[Worker 06]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has missing class preparation view in 3643ms
[Worker 04]   ✔ CourseDetails @group1 › Prepare a student in 272ms
[Worker 04]   ✔ CourseDetails @group1 › Prepare a school, courses, section in 2596ms
[Worker 05]   ✔ CourseDetails @group1 › Prepare a student in 251ms
[Worker 05]   ✔ CourseDetails @group1 › Prepare a school, courses, section in 2360ms
[Worker 06]   ✔ CourseDetails @group1 › Prepare a student in 241ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Login in 4633ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Open Course details in 1053ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":1,"type":"Grading Policy"} in 2ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":2,"type":"Class Policy"} in 1ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":3,"type":"Classroom Materials"} in 2ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":4,"type":"Other Information"} in 2ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials are not present in course detail in 2ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Course description 'has no description set' text in course detail in 519ms
[Worker 06]   ✔ CourseDetails @group1 › Prepare a school, courses, section in 2190ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Login in 4626ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Open Course details in 1091ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":1,"type":"Grading Policy"} in 2ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":2,"type":"Class Policy"} in 2ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":3,"type":"Classroom Materials"} in 1ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":4,"type":"Other Information"} in 1ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials are not present in course detail in 1ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Course description 'has no description set' text in course detail in 543ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Login in 4668ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Open Course details in 1098ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":1,"type":"Grading Policy"} in 4ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":2,"type":"Class Policy"} in 1ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":3,"type":"Classroom Materials"} in 2ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":4,"type":"Other Information"} in 2ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials are not present in course detail in 2ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Course description 'has no description set' text in course detail in 540ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Login in 10664ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Create an iPlanner in 4116ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Login in 12191ms
[Worker 04]   ✔ CourseDetails @group1 › Assign the iPlanner to the section in 336ms
[Worker 04]   ✔ CourseDetails @group1 › Update Course description text in 247ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Open policies tab in 1045ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":1,"type":"Grading Policy"} in 2038ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Login in 12631ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Create an iPlanner in 4335ms
[Worker 05]   ✔ CourseDetails @group1 › Assign the iPlanner to the section in 335ms
[Worker 05]   ✔ CourseDetails @group1 › Update Course description text in 237ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":2,"type":"Class Policy"} in 1976ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Open policies tab in 1084ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":3,"type":"Classroom Materials"} in 2065ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":1,"type":"Grading Policy"} in 2203ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Create an iPlanner in 4289ms
[Worker 06]   ✔ CourseDetails @group1 › Assign the iPlanner to the section in 340ms
[Worker 06]   ✔ CourseDetails @group1 › Update Course description text in 223ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":4,"type":"Other Information"} in 2009ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Open policies tab in 1041ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":2,"type":"Class Policy"} in 2106ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":1,"type":"Grading Policy"} in 2096ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":3,"type":"Classroom Materials"} in 2201ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":2,"type":"Class Policy"} in 2066ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":4,"type":"Other Information"} in 2158ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add an Instructional material of type textbook in 5664ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":3,"type":"Classroom Materials"} in 2146ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":4,"type":"Other Information"} in 2117ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add an Instructional material of type textbook in 6156ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Approve added Policies in 5574ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Course description text in course detail in 2828ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add an Instructional material of type textbook in 5815ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":1,"type":"Grading Policy"} in 1548ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Approve added Policies in 5683ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":2,"type":"Class Policy"} in 1548ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":3,"type":"Classroom Materials"} in 1552ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Course description text in course detail in 2915ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":4,"type":"Other Information"} in 1546ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Approve added Policies in 5628ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials text in course detail in 1038ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":1,"type":"Grading Policy"} in 1625ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":2,"type":"Class Policy"} in 1622ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Course description text in course detail in 2840ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":3,"type":"Classroom Materials"} in 1619ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":1,"type":"Grading Policy"} in 1614ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":4,"type":"Other Information"} in 1620ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":2,"type":"Class Policy"} in 1613ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials text in course detail in 1084ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":3,"type":"Classroom Materials"} in 1616ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":4,"type":"Other Information"} in 1625ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials text in course detail in 1090ms

  OK  | 129 passed   // 2m

real    2m18.414s
user    1m43.318s
sys     0m6.250s

Without test sorting the parallelization per suite works well and the test is fast ✔️ :

mirao@rog:~/workspace/codeceptjs/tests/s4e$ time codeceptjs run-workers 3 dev --by suite --grep "ClassPreparation|CourseDetails"
CodeceptJS v3.7.6 #StandWithUkraine
Running tests in 3 workers...

[Worker 01]   ✔ ClassPreparation @group1 › Get an admin token in 239ms
[Worker 04]   ✔ CourseDetails @group1 › Prepare a student in 242ms
[Worker 01]   ✔ ClassPreparation @group1 › Create a teacher in 692ms
[Worker 02]   ✔ ClassPreparation @group1 › Get an admin token in 244ms
[Worker 05]   ✔ CourseDetails @group1 › Prepare a student in 240ms
[Worker 02]   ✔ ClassPreparation @group1 › Create a teacher in 664ms
[Worker 03]   ✔ ClassPreparation @group1 › Get an admin token in 240ms
[Worker 04]   ✔ CourseDetails @group1 › Prepare a school, courses, section in 2043ms
[Worker 01]   ✔ ClassPreparation @group1 › Prepare a school, school course, section in 2072ms
[Worker 03]   ✔ ClassPreparation @group1 › Create a teacher in 702ms
[Worker 01]   ✔ ClassPreparation @group1 › Create a schedule for the section in 216ms
[Worker 01]   ✔ ClassPreparation @group1 › Create a school calendar in 545ms
[Worker 06]   ✔ CourseDetails @group1 › Prepare a student in 248ms
[Worker 05]   ✔ CourseDetails @group1 › Prepare a school, courses, section in 2090ms
[Worker 02]   ✔ ClassPreparation @group1 › Prepare a school, school course, section in 2084ms
[Worker 02]   ✔ ClassPreparation @group1 › Create a schedule for the section in 191ms
[Worker 02]   ✔ ClassPreparation @group1 › Create a school calendar in 542ms
[Worker 03]   ✔ ClassPreparation @group1 › Prepare a school, school course, section in 2087ms
[Worker 03]   ✔ ClassPreparation @group1 › Create a schedule for the section in 187ms
[Worker 06]   ✔ CourseDetails @group1 › Prepare a school, courses, section in 2039ms
[Worker 03]   ✔ ClassPreparation @group1 › Create a school calendar in 528ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Login in 4505ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Open Course details in 1072ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":1,"type":"Grading Policy"} in 2ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":2,"type":"Class Policy"} in 3ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":3,"type":"Classroom Materials"} in 2ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":4,"type":"Other Information"} in 4ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials are not present in course detail in 2ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Login in 4657ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Course description 'has no description set' text in course detail in 538ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Open Course details in 1098ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":1,"type":"Grading Policy"} in 1ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":2,"type":"Class Policy"} in 2ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":3,"type":"Classroom Materials"} in 3ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":4,"type":"Other Information"} in 1ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials are not present in course detail in 2ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Login in 4759ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Course description 'has no description set' text in course detail in 555ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Open Course details in 1116ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":1,"type":"Grading Policy"} in 2ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":2,"type":"Class Policy"} in 1ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":3,"type":"Classroom Materials"} in 2ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Policy is not present in course detail | {"position":4,"type":"Other Information"} in 1ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials are not present in course detail in 2ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Course description 'has no description set' text in course detail in 542ms
[Worker 01]   ✔ ClassPreparation @group1 › S4W: Login in 11367ms
[Worker 01]   ✔ ClassPreparation @group1 › S4W: Create an autonomous iPlanner in 3150ms
[Worker 02]   ✔ ClassPreparation @group1 › S4W: Login in 15393ms
[Worker 03]   ✔ ClassPreparation @group1 › S4W: Login in 15808ms
[Worker 01]   ✔ ClassPreparation @group1 › S4W: Create a lesson unit and a lesson in 3418ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Login in 14517ms
[Worker 02]   ✔ ClassPreparation @group1 › S4W: Create an autonomous iPlanner in 3316ms
[Worker 03]   ✔ ClassPreparation @group1 › S4W: Create an autonomous iPlanner in 3255ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Login in 16140ms
[Worker 02]   ✔ ClassPreparation @group1 › S4W: Create a lesson unit and a lesson in 3650ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Create an iPlanner in 4114ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Login in 15483ms
[Worker 04]   ✔ CourseDetails @group1 › Assign the iPlanner to the section in 334ms
[Worker 04]   ✔ CourseDetails @group1 › Update Course description text in 325ms
[Worker 03]   ✔ ClassPreparation @group1 › S4W: Create a lesson unit and a lesson in 3506ms
[Worker 01]   ✔ ClassPreparation @group1 › S4W: Add an instrument and set it for class preparation in 6636ms
[Worker 01]   ✔ ClassPreparation @group1 › Assign the iPlanner to the section in 368ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Open policies tab in 1047ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Create an iPlanner in 4461ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":1,"type":"Grading Policy"} in 2132ms
[Worker 05]   ✔ CourseDetails @group1 › Assign the iPlanner to the section in 325ms
[Worker 05]   ✔ CourseDetails @group1 › Update Course description text in 224ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Create an iPlanner in 4424ms
[Worker 06]   ✔ CourseDetails @group1 › Assign the iPlanner to the section in 337ms
[Worker 06]   ✔ CourseDetails @group1 › Update Course description text in 234ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Open policies tab in 1087ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":2,"type":"Class Policy"} in 2050ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Open policies tab in 1064ms
[Worker 02]   ✔ ClassPreparation @group1 › S4W: Add an instrument and set it for class preparation in 7017ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":1,"type":"Grading Policy"} in 2225ms
[Worker 02]   ✔ ClassPreparation @group1 › Assign the iPlanner to the section in 356ms
[Worker 01]   ✔ ClassPreparation @group1 › S4E: Login as a teacher in 6200ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":3,"type":"Classroom Materials"} in 2095ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":1,"type":"Grading Policy"} in 2108ms
[Worker 03]   ✔ ClassPreparation @group1 › S4W: Add an instrument and set it for class preparation in 6988ms
[Worker 03]   ✔ ClassPreparation @group1 › Assign the iPlanner to the section in 365ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":2,"type":"Class Policy"} in 2123ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":4,"type":"Other Information"} in 2051ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":2,"type":"Class Policy"} in 2095ms
[Worker 01]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default class preparation view in 2629ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":3,"type":"Classroom Materials"} in 2221ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":3,"type":"Classroom Materials"} in 2178ms
[Worker 01]   ✔ ClassPreparation @group1 › S4E: Start a class with the lesson in 2115ms
[Worker 02]   ✔ ClassPreparation @group1 › S4E: Login as a teacher in 6517ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":4,"type":"Other Information"} in 2470ms
[Worker 01]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default teacher view in 2148ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add policy | {"position":4,"type":"Other Information"} in 2234ms
[Worker 03]   ✔ ClassPreparation @group1 › S4E: Login as a teacher in 6684ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Add an Instructional material of type textbook in 5759ms
[Worker 01]   ✔ ClassPreparation @group1 › S4W: Disable the instrument for class preparation in 1628ms
[Worker 02]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default class preparation view in 2714ms
[Worker 01]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has missing class preparation view in 1773ms
[Worker 03]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default class preparation view in 2770ms
[Worker 02]   ✔ ClassPreparation @group1 › S4E: Start a class with the lesson in 2167ms
[Worker 03]   ✔ ClassPreparation @group1 › S4E: Start a class with the lesson in 2247ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Add an Instructional material of type textbook in 6172ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Add an Instructional material of type textbook in 5983ms
[Worker 02]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default teacher view in 2169ms
[Worker 04]   ✔ CourseDetails @group1 › S4W: Approve added Policies in 5584ms
[Worker 03]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has default teacher view in 2242ms
[Worker 02]   ✔ ClassPreparation @group1 › S4W: Disable the instrument for class preparation in 1667ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Course description text in course detail in 2904ms
[Worker 03]   ✔ ClassPreparation @group1 › S4W: Disable the instrument for class preparation in 1719ms
[Worker 02]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has missing class preparation view in 1786ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":1,"type":"Grading Policy"} in 1546ms
[Worker 05]   ✔ CourseDetails @group1 › S4W: Approve added Policies in 5719ms
[Worker 06]   ✔ CourseDetails @group1 › S4W: Approve added Policies in 5680ms
[Worker 03]   ✔ ClassPreparation @group1 › S4E: Verify the lesson has missing class preparation view in 2084ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":2,"type":"Class Policy"} in 1553ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":3,"type":"Classroom Materials"} in 1544ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Course description text in course detail in 2922ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Course description text in course detail in 2857ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":4,"type":"Other Information"} in 1548ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":1,"type":"Grading Policy"} in 1626ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":1,"type":"Grading Policy"} in 1648ms
[Worker 04]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials text in course detail in 1043ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":2,"type":"Class Policy"} in 1626ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":2,"type":"Class Policy"} in 1616ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":3,"type":"Classroom Materials"} in 1623ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":3,"type":"Classroom Materials"} in 1647ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":4,"type":"Other Information"} in 1627ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify Policy texts in Course detail | {"position":4,"type":"Other Information"} in 1646ms
[Worker 05]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials text in course detail in 1092ms
[Worker 06]   ✔ CourseDetails @group1 › S4U: Verify the Instructional materials text in course detail in 1107ms

  OK  | 129 passed   // 1m

real    1m20.851s
user    1m50.328s
sys     0m6.696s

[mirao]

Originally reported within #5215 (comment), but it was not caused by that PR. It's caused by #5386

[mirao]

I'm attaching my config from the project where it can be reproduced.

// Load config with sensitive values such as passwords, keys, tokens etc.
process.env["NODE_CONFIG_DIR"] = "../config/";
import authConfig from "config";
import { chromium, webkit, firefox } from "playwright";
import { Browser } from "../steps/Browsers";
import { Dimensions } from "../steps/ScreenImageDimensions";

/** Run only tests tagged by this tag on ACC */
const FILTER_ACC = "@loginScreen";

/** Path to the user's QR code in MJPEG file (a video file format used for streaming of a QR code in Chromium) */
const QR_CODE_USER_MJPEG_PATH = "output/qr_user.mjpeg";

/** Browser window size */
const viewport = {
    width: Dimensions.FULL_HD_WIDTH,
    height: Dimensions.FULL_HD_HEIGHT,
};

/*
 Profile options
 E.g. --profile personal,webkit,window means: Run test in SPORK personal environment on WebKit in windowed mode
 Default values: DEV, chromium, headless mode 
 */
let env, browser, windowMode;
if (process.env.profile) {
    [env, browser, windowMode] = process.env.profile.split(",");
}
// A workaround for the issue that if you omit --profile in the "run-rerun" mode, then `process.env.profile` is a string with the "undefined" value instead of undefined
if (env === "undefined") {
    env = undefined;
}
env = env || "dev"; // undefined or "" means "dev"

/** Browser configuration for run-multiple mode */
const multiBrowserSetting = [
    {
        browser: chromium.name(),
        outputName: chromium.name(),
    },
    {
        browser: webkit.name(),
        outputName: webkit.name(),
    },
    {
        browser: firefox.name(),
        outputName: firefox.name(),
    },
];

export const config: CodeceptJS.MainConfig = {
    qrCodeUserMjpegPath: QR_CODE_USER_MJPEG_PATH,
    env, // Environment, e.g. "dev"
    output: "./output",
    helpers: {
        Common: {
            require: "../helpers/common_helper.ts",
            viewport,
        },
        Playwright: {
            url: ((): string | undefined => {
                switch (env) {
                    default: // DEV environment
                        return authConfig.get("dev.s4eUrl");
                    case "personal":
                        return authConfig.get("personal.s4eUrl");
                    case "acc":
                        return authConfig.get("acc.s4eUrl");
                }
            })(),
            browser: (browser || chromium.name()) as Browser,
            show: windowMode === "window",
            restart: "session",
            timeout: 30000,
            keepBrowserState: true,
            keepCookies: true,
            waitForNavigation: "load",
            waitForTimeout: 30000,
            waitForAction: 500, // Delay (ms) after click
            get chromium() {
                return {
                    args: [
                        "--use-fake-device-for-media-stream",
                        `--use-file-for-fake-video-capture=${QR_CODE_USER_MJPEG_PATH}`,
                        "--remote-debugging-port=9222",
                    ],
                };
            },
            channel: "chromium",
        },
        REST: {
            timeout: 60000,
            prettyPrintJson: true,
        },
        JSONResponse: {},
        ChaiWrapper: {
            require: "codeceptjs-chai",
        },
        CmdHelper: {
            require: "../../node_modules/codeceptjs-cmdhelper",
        },
        Mochawesome: {
            uniqueScreenshotNames: true,
        },
        PixelmatchHelper: {
            require: "codeceptjs-pixelmatchhelper",
            dirExpected: "./screenshots/base/",
            dirDiff: "./screenshots/diff/",
        },
        S4E: {
            require: "./S4E_helper.ts",
            /** Allow actions after each step */
            afterStepFeatures: {
                visualTesting: false,
                errorLogCheck: true,
            },
            urlS4W: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return authConfig.get("dev.s4wUrl");
                    case "personal":
                        return authConfig.get("personal.s4wUrl");
                }
            })(),
        },
        BrowserWebSocket: {
            require: "../helpers/browser_websocket_helper.ts",
        },
        SPORK_Server: {
            require: "../helpers/spork_server_helper.ts",
            url: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return authConfig.get("dev.sporkServer");
                    case "personal":
                        return authConfig.get("personal.sporkServer");
                }
            })(),
        },
        S4A_Server: {
            require: "../helpers/s4a_server_helper.ts",
            url: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return authConfig.get("dev.s4aServer");
                    case "personal":
                        return authConfig.get("personal.s4aServer");
                }
            })(),
        },
        Firebase: {
            require: "../helpers/firebase_helper.ts",
            googleApiUrl: authConfig.get("common.googleApiUrl"),
            firebaseStorageUrl: authConfig.get("common.firebaseStorageUrl"),
            firebaseStorageBucket: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return `${authConfig.get(
                            "dev.firebaseProjectId"
                        )}.${authConfig.get(
                            "common.firebaseStorageBucketDomain"
                        )}`;
                    case "personal":
                        return `${authConfig.get(
                            "personal.firebaseProjectId"
                        )}.${authConfig.get(
                            "common.firebaseStorageBucketDomain"
                        )}`;
                }
            })(),
            firebaseDbDomain: authConfig.get("common.firebaseDbDomain"),
            firebaseRealTimeDbHost: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return authConfig.get("dev.firebaseRealTimeDbHost");
                    case "personal":
                        return authConfig.get(
                            "personal.firebaseRealTimeDbHost"
                        );
                }
            })(),
            firebaseWebApiKey: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return authConfig.get("dev.firebaseWebApiKey");
                    case "personal":
                        return authConfig.get("personal.firebaseWebApiKey");
                }
            })(),
            firebaseLegacyToken: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return authConfig.get("dev.firebaseLegacyToken");
                    case "personal":
                        return authConfig.get("personal.firebaseLegacyToken");
                }
            })(),
        },
        MySQL_DB: {
            // Custom DB helper
            require: "../helpers/mysql_db_helper.ts",
            spork_connection: ((): string => {
                switch (env) {
                    default: // DEV environment
                        return `mysql://${authConfig.get(
                            "dev.sporkDbUser"
                        )}:${authConfig.get(
                            "dev.sporkDbPassword"
                        )}@${authConfig.get("dev.sporkDbUrl")}`;
                    case "personal":
                        return `mysql://${authConfig.get(
                            "personal.sporkDbUser"
                        )}:${authConfig.get(
                            "personal.sporkDbPassword"
                        )}@${authConfig.get("personal.sporkDbUrl")}`;
                }
            })(),
        },
    },
    mocha: {
        reporterOptions: {
            "codeceptjs-cli-reporter": {
                stdout: "-",
                options: {
                    verbose: true,
                    debug: true,
                    steps: true,
                },
            },
            mochawesome: {
                stdout: "./output/console.log",
                options: {
                    reportDir: "./output",
                    reportFilename: "report",
                },
            },
            "mocha-junit-reporter": {
                stdout: "./output/console.log",
                options: {
                    mochaFile: "./output/result.xml",
                    jenkinsMode: true,
                    attachments: true, //add screenshot for a failed test
                },
            },
        },
    },
    plugins: {
        pauseOnFail: {
            //Enable it manually on each run via -p option
        },
        retryFailedStep: {
            enabled: true,
            retries: 10,
            // Don't retry the following steps when calling step from test, i.e. `I.dontSeeErrorBrowserConsole()`
            // _afterStep() in helper doesn't retry the steps even without that setting
            ignoredSteps: ["dontSeeErrorBrowserConsole", "assert*"],
        },
        screenshotOnFail: {
            enabled: true,
            uniqueScreenshotNames: true,
        },
        stepByStepReport: {
            enabled: false,
            deleteSuccessful: false,
        },
        customLocator: {
            enabled: true,
            prefix: "#",
            attribute: "data-testid",
        },
    },
    tests: "./*_test.ts",
    name: "s4e",
    grep: env === "acc" ? FILTER_ACC : "",
    multiple: {
        dev: {
            browsers: multiBrowserSetting,
        },
        acc: {
            grep: FILTER_ACC,
            browsers: multiBrowserSetting,
        },
        personal: {
            browsers: multiBrowserSetting,
        },
    },
    rerun: {
        // how many times all tests should pass
        minSuccess: 20,

        // how many times to try to rerun all tests
        maxReruns: 20,
    },
    timeout: 600,
    require: ["tsconfig-paths/register"],
};