fix(scale-up): prevent negative TotalTargetCapacity when runners exceed maximum

[shivdesh]

Problem

When pool and scale-up lambdas run concurrently, currentRunners can temporarily exceed maximumRunners. This causes the calculation maximumRunners - currentRunners to produce a negative value, which is then passed to the EC2 CreateFleet API.

Error observed in CloudWatch logs:

InvalidTargetCapacitySpecification: TotalTargetCapacity should not be negative.

Root cause:

The scale-up lambda calculates:

const newRunners = Math.min(scaleUp, maximumRunners - currentRunners);

When currentRunners > maximumRunners (e.g., 55 runners with max 50), this produces:

newRunners = Math.min(1, 50 - 55) = Math.min(1, -5) = -5

This negative value is passed to CreateFleet API which rejects it.

Solution

Wrap the calculation with Math.max(0, ...) to ensure we never attempt to create a negative number of runners:

const newRunners = Math.max(0, Math.min(scaleUp, maximumRunners - currentRunners));

Changes

• scale-up.ts: Added Math.max(0, ...) guard to prevent negative runner count
• scale-up.test.ts: Added test case for race condition scenario

Testing

• Added unit test that simulates 10 current runners with max of 5
• Verified no createRunner call is made in this scenario
• All existing tests pass

[Copilot]

Pull request overview

Fixes an edge case in the control-plane runner scaling logic where concurrent pool/scale-up activity can temporarily push currentRunners above maximumRunners, leading to negative target capacity being sent to EC2. Also adds retry handling to improve resilience when de-registering runners from GitHub during scale-down.

Changes:

• Guard newRunners calculation in scale-up.ts with Math.max(0, ...) to prevent negative capacity.
• Add a unit test covering the “current runners exceed maximum” race condition scenario.
• Add retry-with-backoff behavior for GitHub runner deletion in scale-down.ts, plus tests for transient 5xx failures.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
lambdas/functions/control-plane/src/scale-runners/scale-up.ts	Prevents negative runner counts from being passed into fleet creation.
lambdas/functions/control-plane/src/scale-runners/scale-up.test.ts	Adds coverage for the race-condition scenario where current runners exceed max.
lambdas/functions/control-plane/src/scale-runners/scale-down.ts	Introduces retry logic for GitHub runner de-registration to handle transient API errors.
lambdas/functions/control-plane/src/scale-runners/scale-down.test.ts	Adds tests validating retry behavior and non-termination when retries are exhausted.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.