feat(ray_tune): difference stopper

[michael-johnston]

This PR add a new stopper, BayesianDifferenceStopper, to ray tune operator.

This stopper enables stopping sample when the mean difference in two metrics is determined to be above or below a threshold with a target certainty.

Use cases including determining if a model is drifting i.e. if the mean difference is above or below a tolerance threshold given new data; and detecting performance regressions i.e. if given a new software version software performance has changed substantially (beyond a threshold)

[AlessandroPomponio]

I will need to check again tomorrow, but these are some issues I see

[michael-johnston]

@AlessandroPomponio hold off on review as I noticed a more fundamental issue I need to fix

will convert to draft and ping you when ready

[DRL-NextGen]

Checks Summary

Last run: 2026-01-16T11:23:13.286Z

Code Risk Analyzer vulnerability scan found 2 vulnerabilities:

Severity	Identifier	Package	Details	Fix
🔷 Medium	CVE-2026-22773	vllm	vLLM is vulnerable to DoS in Idefics3 vision models via image payload with ambiguous dimensions GHSA-grg2-63fw-f2qr vllm:0.11.1	0.12.0
◻ Unknown	CVE-2025-53000	nbconvert	nbconvert has an uncontrolled search path that leads to unauthorized code execution on Windows GHSA-xm59-rqc7-hhvf nbconvert:7.16.6->ado-core:1.3.3	>7.16.6

Mend Unified Agent vulnerability scan found 1 vulnerabilities:

Severity	Identifier	Package	Details	Fix
🔺 High	CVE-2025-53000	nbconvert-7.16.6-py3-none-any.whl	The nbconvert tool, jupyter nbconvert, converts Jupyter notebooks to various other formats via Jinja... The nbconvert tool, jupyter nbconvert, converts Jupyter notebooks to various other formats via Jinja templates. Versions of nbconvert up to and including 7.16.6 on Windows have a vulnerability in which converting a notebook containing SVG output to a PDF results in unauthorized code execution. Specifically, a third party can create a "inkscape.bat" file that defines a Windows batch script, capable of arbitrary code execution. When a user runs "jupyter nbconvert --to pdf" on a notebook containing SVG output to a PDF on a Windows platform from this directory, the "inkscape.bat" file is run unexpectedly. As of time of publication, no known patches exist.	Not Available

Changes addressed

[michael-johnston]

@VassilisVassiliadis suggested operation for testing with sfttrainer. Change fields as desired.

metadata:
  description: "Perform latin hypercube sampling with difference stopper for space using sfttrainer lora benchmark experiment"
  name: "lhc-difference-sfttrainer-lora"
operation:
  module:
    operatorName: "ray_tune"
    operationType: "search"
  parameters:
    runtimeConfig:
      stop:
      - name: "BayesianMetricDifferenceStopper"
        keywordParams:
          metric_a: "finetune_lora_benchmark-v1.0.0-fms_hf_tuning_version.2.6.0-dataset_tokens_per_second_per_gpu"  # v1 measurement
          metric_b: "finetune_lora_benchmark-v1.0.0-fms_hf_tuning_version.3.0.0-dataset_tokens_per_second_per_gpu"  # v2 measurement
          threshold: 100                  # Stop when we know |v1-v2| > or < 100 with target probability
          target_probability: 0.95        # 95% confidence
          min_samples: 10                 # Wait for 10 trials minimum
    orchestratorConfig:
      metric_format: "observed" # We need to use observed property value as the target property id is the same for both experiment versions
    tuneConfig:
      metric: "finetune_lora_benchmark-v1.0.0-fms_hf_tuning_version.2.6.0-dataset_tokens_per_second_per_gpu" #ray tune needs primary metric to track
      max_concurrent_trials: 1 # This is set for debugging. Increase if you want multiple measurements at once.
      mode: min 
      num_samples: 32
      search_alg:
        name: lhu_sampler
spaces:
  - space-60b5c0-12e5dd

Changes addressed

[michael-johnston]

@VassilisVassiliadis We will wait on your update that above YAML works before merging.

[VassilisVassiliadis]

I'll reply here when my test is over.