[Draft] feat(vardiff): add in-process simulation framework + baseline regression tests

[gimballock]

Adds a deterministic in-process simulation framework that characterizes
any Vardiff implementation across the operational rate range, and
commits the current algorithm's measurements as a baseline for automated
regression testing.

The "vardiff fires too often on noise" / "vardiff doesn't react fast
enough" conversations have circled the same issues (#396 and adjacent)
without a way to settle questions empirically. This PR adds the missing
infrastructure: any future proposal can now produce a quantitative
delta against a fixed reference.

The finding that motivates this

Before the framework, "is the algorithm too noisy or too sluggish?" was
a matter of opinion. With it, the question is a table:

Reaction sensitivity to a -50% step change (probability of firing
within 5 minutes after the change):

share/min	sensitivity
6	0.70
12	0.55
30	0.33
60	0.16
120	0.09

Higher share rates produce less responsive algorithms — counter to
expectations. The mechanism is mechanical (post-convergence delta_time
grows indefinitely, diluting the post-step signal in the cumulative
window) and surfaced clearly in the data despite taking months of
deployment observation to half-articulate. Full numbers and analysis
in vardiff_baseline.md.

This isn't a fix. It's the measurement that lets the fix be evaluated.

What's in the PR

3 commits, ~2000 LOC plus baseline data:

1. feat(vardiff): inject Clock trait + add_shares trait method
   Minimum API additions to channels_sv2 for testability and
   simulation performance. Production behavior unchanged — existing
   constructors default to SystemClock, the new trait method has a
   default implementation that keeps existing impls compiling.

2. feat(vardiff_sim): in-process simulation framework
   New crate at sv2/channels-sv2/sim/. Per-tick Poisson share
   sampling, five behavioral metrics with percentile distributions,
   50-cell parameterized sweep (5 share rates × 10 scenarios), CLI
   binary for baseline generation, regression test asserting against
   a committed baseline.

3. data(vardiff_sim): design doc + baseline characterization
   The design proposal documenting metric definitions and tolerance
   policy, plus the measured baseline as both TOML (consumed by the
   regression test) and Markdown (for human review).

What the framework measures

Five behavioral attributes, each as a distribution across 1000
independent trials per cell:

Metric	Better is	What it tells you
Convergence time	Smaller	How fast the algorithm settles after cold start
Settled accuracy	Smaller	How close to truth the algorithm lands
Steady-state jitter	Smaller	How often it fires on noise post-settle
Reaction time	Smaller	How fast it responds to genuine load changes
Reaction sensitivity	≈ 1 for real Δ, ≈ 0 for noise	Whether it distinguishes signal from noise

Per-metric tolerances are asserted automatically against the checked-in
baseline. Failed assertions identify the cell and metric with specific
baseline-vs-current numbers. Mid-range Δ values (10-25%) are reported
but not asserted — that's where legitimate algorithmic tradeoffs live
and a reviewer should judge by looking at the full delta.

How to run

From sv2/channels-sv2/sim/:

# Fast unit tests (~1 second)
cargo test

# Generate a fresh baseline (~5-15 seconds)
cargo run --release --bin generate-baseline

# Run the slow regression test (~5-15 seconds; #[ignore]-d by default)
cargo test --release --lib -- --ignored

What this enables

For any future vardiff proposal:

1. Implement the new algorithm as a Vardiff impl
2. cargo run --release --bin generate-baseline to produce comparable
   measurements
3. Diff against the committed baseline
4. Make the case with numbers

No more "I think this is better." Instead "this changes p50 jitter from
X to Y at 12 spm, at the cost of p90 reaction time going from A to B."

Where to look in this PR

• Design proposal (architecture, metric definitions, tolerance
  rationale): sv2/channels-sv2/sim/VARDIFF_SIMULATION_FRAMEWORK.md
• Crate README (usage, output interpretation, baseline-update
  workflow): sv2/channels-sv2/sim/README.md
• The current algorithm's measured baseline:
  sv2/channels-sv2/sim/vardiff_baseline.md

What this PR is NOT

• Not an algorithm change. VardiffState behavior is unchanged.
  The only public-API additions are Vardiff::add_shares (with a
  default impl) and the Clock trait. Production code defaults to
  SystemClock and behaves identically to before.
• Not a recommendation about share rate defaults. The baseline
  data suggests 12-30 spm is the operational sweet spot, but this
  PR doesn't touch any defaults.
• Not a CI workflow. The regression test works locally but needs
  a GitHub Action to be a true CI gate. Follow-up.

Open follow-ups

• Wire cargo test --release --lib -- --ignored into CI on PRs
  touching vardiff/* or the sim crate.
• Bump channels_sv2 5.0.0 → 5.1.0 once the workspace lockfile
  situation allows (the trait-method addition is technically a
  minor-version semver change). TODO comment in Cargo.toml tracks
  this.
• Investigate the reactivity-degrades-with-rate finding. The framework
  surfaces the problem; fixing it is a separate proposal that this
  framework will be the right tool to evaluate.

Test plan

• cargo test -p channels_sv2 --lib vardiff — 17 tests, all pass
• cargo test from sv2/channels-sv2/sim/ — 53 fast unit tests
• cargo test --release --lib -- --ignored from sim/ — slow
  regression test passes against committed baseline
• cargo run --release --bin generate-baseline — reproduces the
  committed vardiff_baseline.toml byte-for-byte at the same seed

[gimballock]

The code is cheap and only meant to demonstrate the feasibility, but the concept ack revolves around these points imo:

• We can play dice with share-received events to simulate running the vardiff algorithms over arbitrary ranges of time. But we need to mock SystemTime::now() and add a way to bulk add new shares.
• With fake time simulations we can do large scale vardiff trials of whatever metrics we want and contrast against correlated attributes like target shares-per-minute.
  • I was interested in convergence time, stable-state jitter, and convergence accuracy
  • But responsiveness to external change is also a key capability, (how fast to adjust to a 50% spike/dip in hashrate)
• With this compilation of reproducible test results compiled into a profile we can use integration tests to lock in established performance thresholds and ratchet up the expectations if we find better algorithms.

[gimballock]

The first row here shows results of the convergence time test for the the default case (6 spm).
The convergence times are between 10 and 25 minutes, with total failures to converge (w/in quiet_window_secs of simulated time) occurring 17% of the time!

the next few rows describe the results for faster share rates. the most extreme times (25m reduces to 15m) and the total failure cases generally disappear around 30 spm.

[gimballock]

These two metrics (proximity to true hashrate and post-converged adjustments) show a similar trend,
Lots of undesired behavior in the extreme cases (top 10%, top 1%) of 6 shares/min case that is alleviated at higher share rates.

[gimballock]

These tables show how long it takes for vardiff to respond to an unexpected change in hashrate. Where the changes are to either increase or decrease by proportional amounts anywhere from 5% to 50%.

The first table specifically looks at a 50% draw down showing that a full 30% of the time vardiff fails to adjust after 5 min. The next few rows show that the situation worsens at higher share rates, at 120 spm 91% of the trials failed to adjust after 5m.

The second table shows that this effect is basically the same for hashrate changes in the opposite direction and also that changes of lesser magnitude respond much more quickly.

[gimballock]

Convergence rate: Must be no more than 1% slower than the baseline convergence time
Convergence p90: The slowest 10% convergence times must be within 10% of the baseline's convergence time
Settled accuracy: must be within 15% of baseline's accuracy for the slowest 50% / 10%
Jitter p50/p95: must be within 2% and 25% of baseline
...etc.

You see the pattern, there are lots of magic thresholds in this portion of the code that are arbitrarily chosen at this point and fair game for analysis.

[adammwest]

after some optimization I got
https://github.com/adammwest/stratum/blob/feat/vardiff_kalman/sv2/channels-sv2/sim/vardiff_best.md
with the Bayesian model

Method	Result
Bayesian model	Good, Best result
Kalman filters	Good
Jurik moving averages	Good but artifacts on hashrate changes
Thompson sampling	Bad

[gimballock]

after some optimization I got https://github.com/adammwest/stratum/blob/feat/vardiff_kalman/sv2/channels-sv2/sim/vardiff_best.md with the Bayesian model
Method Result
Bayesian model Good, Best result
Kalman filters Good
Jurik moving averages Good but artifacts on hashrate changes
Thompson sampling Bad

I'm so excited to see people other people nerding out on vardiff with me! Thank you!

A couple things I noticed in your results, the 2m convergence time is impressive but your response to a 50% hashrate drop only succeeds in readjusting 4.4% of the time. I'm not sure how best to balance those two metrics but probably not one at the expense of the other.

[adammwest]

Some learning's I had @gimballock

• This task is hard, I think the current implementation is optimized to a degree.
• the most critical thing is the fitness, currently there are many metrics, how you combine all of them into a final value is what determines the goodness for any algorithm, as its a summary there are ways to game it.
• Use every value in the toml file, if you dont those values will naturally will degrade.
• There are many ways to combine many numbers which lead to slightly better and slightly worse performance.
  one thing I did which helped was to separate fitness into 2 categories improvement and regression, even better separating these per group e.g stable,coldstart ,... then you can decompose the value.
• Normalize each metric/group otherwise the more numerous or larger numbers will be the focus.
• There are many cases where you can get a good score, but the fitness prefers optimizing 1 variable or a set of variables at the expense of others.
• For grid parameter sweeps, they usually discretize the domain so you are bounded in improvement only by dimension range and amount of queries. so you need to constantly increase queries or shrink ranges. usually you are limited due to time. For this reason I prefer random restart hill climbing I find is generally pretty good when you don't make assumptions about the data.
• If you have too many parameters to optimize you can over fit, and end up just gaming the test

[gimballock]

Thanks for these insights @adammwest — especially the point about fitness decomposition and normalization. A lot of what you're describing matches the evolution I've gone through on this PR, so let me give a timeline of
how the approach has matured:

Phase 1: Basic metrics + simulation harness

Initially I focused on three metrics I thought were important: convergence time, jitter, and accuracy. These were evaluated via a time-compressed simulation that replays a synthetic share stream through the vardiff
algorithm. This gave us reproducible, large-scale trials (50 cells × 1000 trials) against correlated attributes like target shares-per-minute.

Phase 2: Decomposed pipeline model

I wanted to make algorithm search more systematic, so I decomposed "a vardiff algorithm" into four independent, replaceable components: estimator, statistic, boundary, and decision rule. The idea was to mix-and-match
implementations at each slot for the best composite.

This model worked well for the classic algorithm, the parametric variant, and the EWMA approach. But when I tried to embed a Bayesian model, it broke down — the components aren't truly independent. There's a sequential
data flow: the estimator needs to communicate its belief to the boundary ("should we respond?") and to the update rule. Additionally, since vardiff triggers on a timer rather than on share arrival, the decision rule needs
to call back to the estimator to update state when adjustments occur. I also dropped the "statistic" component as it wasn't pulling its weight.

The resulting three-stage pipeline (Estimator → Boundary → UpdateRule) is what's in this PR. It successfully hosts the classic algorithm, EWMA, AdaCUSUM, and could host a Bayesian approach.

Phase 3: Aggregate fitness metric

To your point about "how you combine all metrics into a final value" — we now have a configurable aggregate metric that allows weighting across the underlying measurements. This addresses exactly the gaming concern you
raised: rather than optimizing one metric at the expense of others, we can define a weighted composite that represents our desired tradeoff. The regression baseline locks in the full vector of metrics so we catch
regressions in any dimension, not just the aggregate.

Your suggestion to separate fitness into improvement vs. regression categories per scenario group (stable, coldstart, reaction) is a good one. Currently the regression test does compare per-cell, so a coldstart regression
can't hide behind a stable-state improvement, but making this more explicit in the scoring would help.

Phase 4: Realistic operating conditions

After discussions with hardware engineers, I retuned the test scenarios to realistic share rates (2–30 spm instead of the earlier 6–120 range). The engineers confirmed that responding to partial hardware failures and
network slowdowns on an established channel is valuable functionality — even though in practice many hashrate changes currently cause miner reconnections (which resets vardiff anyway). We've been doing live testing with
physical miners on testnet4 and confirmed this pattern: when vardiff ramps difficulty too aggressively, it can interact with firmware timeout behaviors in ways that force reconnections, making reactivity testing harder
than expected.

Current direction

I've backed off from prioritizing convergence speed after seeing overcorrection in practice. The current focus is on:

• Stability under steady-state (minimal oscillation once converged)
• Reasonable reactivity (detect genuine changes within 2–3 retarget windows, not 1)
• Asymmetric cost awareness — difficulty increases are more disruptive than decreases. An overshoot upward causes difficulty-too-low share rejections (wasted miner work), while an undershoot downward just means slightly
  more shares than optimal (cheap). The AsymmetricCusumBoundary encodes this: it requires stronger evidence before raising difficulty than lowering it. We can now actually measure the impact via the share-rejection metrics
  (shares_rejected_total{reason="difficulty-too-low"}) that were recently added to the pool's monitoring (sv2-apps PR Docs: Channel Factory #491).

On your point about normalization: agreed, and the per-metric tolerance budgets in the regression test (absolute slack + optional multiplicative slack) are our current mechanism for this. Open to suggestions on better
normalization approaches