Wait for disk utilization info cache to be populated on controllers during startup

[mluvin-stripe]

Problem

When using the “pause ingestion based on resource utilization” feature (docs), upon restart, controllers initially don’t have their cache of server disk utilization information populated until the ResourceUtilizationChecker periodic task runs. There’s a config controller.resource.utilization.checker.initial.delay that we can set to zero seconds to kick off populating the cache immediately, but the controller could still start serving requests before the checker finishes populating the cache since the controller doesn’t wait for the checker to finish before marking itself as ready.

This is a problem for minion-based offline segment generation (code) and offline segment uploads (new feature proposed in #17557), since the disk utilization check will return UNDETERMINED if the controller’s disk utilization cache isn’t yet populated – so the segment creation/upload is allowed to proceed, even if the disk threshold has already been breached.

Solution

I propose adding an opt-in config controller.resource.utilization.checker.waitDuringStartup that ensures the disk utilization cache is populated before marking the controller as ready. This way, the controller is immediately ready to correctly reject segment creation/upload requests after starting up.

I was thinking of adding another serviceStatusCallback (like this one)that checks if the disk utilization cache has been populated yet, and doesn’t return GOOD until it’s populated.

[mluvin-stripe]

@Jackie-Jiang revisiting this after our meeting last week. Zooming out, I think we were discussing two different problems:

1. Preventing segment uploads from succeeding on a controller when it has UNDETERMINED status for the disk utilization check, while other controllers have already started denying uploads due to high disk usage (i.e. incorrectly failing open)
2. Ensuring that the controller’s disk utilization cache is populated before running the disk utilization check

For (2), we discussed potentially updating the controller's disk utilization cache synchronously during each segment upload API call (probably with some TTL). The concern I have with this approach is that we’ve seen the disk utilization checker task (which updates the controller’s disk utilization cache) take up to 3 minutes to run in the worst case (but typically < 500ms avg case), and the segment upload HTTP request will timeout by then. I think a ~20s timeout on disk utilization checker is reasonable, but then we run back into the problem described in (1) above, with an UNDETERMINED checker status.

To address (1), I was thinking we could implement something similar to what the realtime ingestion pausing does: have a field in the segment’s ideal state that marks whether to “pause” consumption (realtime code), which for offline tables means to reject all future segment uploads. This is making the check “sticky.” The first controller to recognize that disk utilization has been exceeded will mark the field in the offline segment’s ideal state, so all future segment uploads will correctly fail if the disk utilization check is UNDETERMINED (this is what the realtime code does).

I think making the disk utilization check “sticky” is more useful than populating the disk utilization cache synchronously, if I had to just choose one. But both would be ideal. Let me know what you think.