kill persistent processes

[leej3]

Description

I am working with a misbehaving test suite that hangs. "retry" has worked very nicely to avoid failures due to this hanging. One issue that I have noticed is that occasionally a pytest session fails to respond to the SIGTERM signal. When this happens two child processes run at the same time and make a bit of a mess. This PR proposes using SIGKILL to make sure the child process exits.

Testing

• What tests were added?
  • To emulate an unresponsive test session I have added an integration test where the child process ignores interrupts to ensure that it is killed appropriately

[leej3]

It might make sense to kill the whole process group in case there are forked processes. ...ignore, I missed the use of tree-kill...

[leej3]

I've made another attempt at this. My js isn't so good so bear with me!

My understanding is that the test suite is ignoring the SIGTERM, here I propose escalating with a SIGINT, and then using SIGKILL to make sure that the retry of the command does not have the old one hanging around.

Apart from addressing child process that ignore SIGTERM this would also address situations where the clean up is just too slow and a user does not wish to wait longer.

I think my test is now better formulated to timeout at 5 mins so as not to waste CI resources.