Cosmic replicaset Kubernetes API client large scale issue

[wanlonghenry]

Summary
This PR enhances the efficiency and reliability of Kubernetes resource fetching and processing. It adds support for streaming and parsing compressed (gzip) JSON payloads, improves continuation token management, and adds extra safeguards for custom metrics processing. The changes primarily affect Kubernetes API client handling, pod inventory processing, and utility checks for custom metrics enablement.

source/plugins/ruby/in_kube_podinventory.rb
Adds a check for the availability of custom metrics via CustomMetricsUtils before pushing mdmPodRecord items.
Prevents caching or pushing pod records for custom metrics when they are not enabled or available.

source/plugins/ruby/KubernetesApiClient.rb
Refactors getResourcesAndContinuationTokenV2 to stream and efficiently parse API responses, supporting both compressed (gzip) and uncompressed JSON payloads.
Adds logic to handle large API payloads by parsing them on-the-fly using Yajl for reduced memory usage, while smaller payloads are parsed fully in-memory.
Improves error handling, fallback mechanisms, resource usage accounting, and logging for API calls.
Cleanly manages continuation token extraction and propagates it in all parsing scenarios.
Adds new dependencies: yajl, zlib, and stringio for streaming and decompression support.

source/plugins/ruby/CustomMetricsUtils.rb
Enhances the check_custom_metrics_availability function: now disables custom metrics if the relevant environment variable is missing or set to an empty string, in addition to 'false'.

Scenaiors:

PATH 1: full_gzip
Condition: Response has Content-Encoding: gzip and Content-Length ≤ 256 KB.

PATH 2: full_plain
Condition: Response has no gzip (Content-Encoding: not set) and Content-Length ≤ 256 KB.

PATH 3: stream_gzip Condition: Response has Content-Encoding: gzip and size is larger than 256 KB (or no Content-Length).

PATH 4: stream_plain
Condition: Response is plain (no gzip) and larger than 256 KB (or no Content-Length).

[zanejohnson-azure]

Summary This PR enhances the efficiency and reliability of Kubernetes resource fetching and processing. It adds support for streaming and parsing compressed (gzip) JSON payloads, improves continuation token management, and adds extra safeguards for custom metrics processing. The changes primarily affect Kubernetes API client handling, pod inventory processing, and utility checks for custom metrics enablement.

source/plugins/ruby/in_kube_podinventory.rb Adds a check for the availability of custom metrics via CustomMetricsUtils before pushing mdmPodRecord items. Prevents caching or pushing pod records for custom metrics when they are not enabled or available.

source/plugins/ruby/KubernetesApiClient.rb Refactors getResourcesAndContinuationTokenV2 to stream and efficiently parse API responses, supporting both compressed (gzip) and uncompressed JSON payloads. Adds logic to handle large API payloads by parsing them on-the-fly using Yajl for reduced memory usage, while smaller payloads are parsed fully in-memory. Improves error handling, fallback mechanisms, resource usage accounting, and logging for API calls. Cleanly manages continuation token extraction and propagates it in all parsing scenarios. Adds new dependencies: yajl, zlib, and stringio for streaming and decompression support.

source/plugins/ruby/CustomMetricsUtils.rb Enhances the check_custom_metrics_availability function: now disables custom metrics if the relevant environment variable is missing or set to an empty string, in addition to 'false'.

Testing:

could add description how was the test done, and what was verified in this test?